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EXECUTIVE  SUMMARY 


Overview 

This  first  summative  evaluation  of  the  Collegiate 
Training  Initiative  for  Air  Traffic  Control  Specialists 
(CTI/ATCS)  focused  on  the  progress  of  Minnesota 
Air  Traffic  Control  Training  Center  (MnATCTC) 
graduates  in  en  route  field  training.  The  evaluation 
was  framed  by  two  questions; 

•  How  are  MnATCTC  program  graduates  progress¬ 
ing  in  en  route  field  training  relative  to  a  comparison 
group  of  FAA  Academy  graduates? 

•  What  are  the  costs  and  benefits  of  the  MnATCTC 
program  for  the  Federal  Aviation  Administration 
(FAA)? 

Method 

The  evaluation  compared  1 36  MnATCTC  gradu¬ 
ates  hired  between  1990  and  1993  with  157  FAA 
Academy  graduates  who  entered  en  route  field  train¬ 
ing  in  May  and  June  1991.  The  groups  were  com¬ 
pared  on  4  measures: 

•  Diversity 

•  Progress  in  en  route  field  training  at  first  assigned  en 
route  facilities 

•  Attrition  from  first  assigned  en  route  facilities 
•Performance  ratings  at  first  assigned  en  route 

facilities 

Supervisory,  on-the-job  training  instructor  (OJT- 
I),  and  self-evaluations  were  collected  in  spring  1994 
by  mail.  Field  training  and  attrition  measures  were 
extracted  from  FAA  and  Civil  Aeromedical  Institute 
(CAMI)  databases  in  June  1995. 

Direct  and  indirect  costs  associated  with  the 
MnATCTC  program  were  estimated  and  projected 
through  the  year  2003  to  compare  per-hire  costs  with 
FAA  Academy  per-controller  costs.  Four  classes  of 
monetary  benefits  to  the  agency  were  identified:  (1) 
avoided  screening  costs;  (2)  avoided  Academy  train¬ 


ing  costs;  (3)  avoided  performance  verification  (PV) 
costs;  and  (4)  savings  from  reduced  time-to-FPL  for 
MnATCTC  graduates.  Program  benefits  were  esti¬ 
mated  and  projected  through  the  year  2003  to  con¬ 
duct  a  benefit-to-cost  analysis  for  the  MnATCTC 
program  as  a  model  for  future  CTI/ATCS  cost- 
benefit  analyses. 

Results 

Diversity 

There  were  significantly  more  women  in  the 
MnATCTC  group  (40%)  than  in  the  FAA  Academy 
sample  (17%).  There  was  no  difference  in  the  repre¬ 
sentation  of  minorities. 

Field  training  progress 

Training  phases.  MnATCTC  graduates  did  better 
on  six  measures  of  training  performance,  relative  to 
historical  facility  averages:  number  of  days  to  com¬ 
plete  Phase  VII;  number  of  days  to  complete  Phase 
IX  (raw  and  adjusted  for  the  number  of  sectors  for 
which  training  was  provided);  number  of  days  in 
Phase  X  (raw  and  adjusted  for  number  of  sectors  for 
which  training  was  provided);  and  number  of  days  in 
Phase  XI.  FAA  Academy  graduates  did  better  than 
MnATCTC  graduates  on  three  measures  of  training 
performance,  relative  to  historical  facility  averages: 
indicator  of  performance  (IP)  ratings  in  Phase  VI; 
number  of  days  in  Phase  VIII;  and  number  of  hours 
of  on-the-job  training  (OJT),  adjusted  for  number  of 
sectors,  in  Phase  X. 

Time  to  FPL.  Just  23  of  the  MnATCTC  graduates 
(17%)  had  reached  FPL  at  the  first  assigned  field 
facility  as  of  June  1995,  in  contrast  to  108  of  the  FAA 
Academy  graduates  (68.8%).  There  was  no  signifi¬ 
cant  difference  in  the  average  times  to  FPL  between 
the  groups.  Survival  analysis  found  no  significant 
differences  between  the  groups  in  accession  to  FPL 
over  time. 
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Attrition 

Ten  developmentals  had  attrited  at  the  first  as¬ 
signed  facility  from  the  MnATCTC  group,  com¬ 
pared  with  six  FAA  Academy  graduates.  The  attrition 
rates  were  not  statistically  different.  Significantly 
more  MnATCTC  graduates  {N  =  7)  had  changed 
facilities  or  options  than  FAA  Academy  graduates  {N = 
1;  Z  =  1.99,  p  <  .05).  No  significant  differences 
between  groups  in  attrition  over  time  were  found  by 
survival  analysis. 

Performance  ratings 

Supervisors  rated  controllers  on  items  represent¬ 
ing  five  dimensions:  (a)  technical  skill;  (b)  technical 
knowledge;  (c)  teamwork;  (d)  degree  to  which  the 
developmental  was  accepted  by  others  in  the  facility; 
and  (e)  global  assessment  of  potential  to  succeed  in 
the  ATCS  occupation.  MnATCTC  graduates  were 
rated  significantly  lower  than  FAA  Academy  gradu¬ 
ates  on  all  five  dimensions.  In  particular,  the  average 
rating  on  potential  to  succeed  in  the  occupation  of  81 
for  MnATCTC  graduates  was  significantly  lower  than 
the  average  rating  of  86  for  FAA  Academy  graduates 
(on  a  40  to  100  scale). 

Controllers  completed  a  self-evaluation  using  the 
same  instrument  as  supervisors.  MnATCTC  gradu¬ 
ates  rated  themselves  significantly  lower  on  team¬ 
work,  technical  skill,  technical  knowledge,  and 
potential  to  succeed  in  the  ATCS  occupation  than 
did  FAA  Academy  graduates.  MnATCTC  graduates 
also  felt  significantly  less  well  accepted  at  the  facility 
than  FAA  Academy  graduates. 

Program  costs  and  benefits 

The  comparison  of  MnATCTC  per-hire  costs 
with  FAA  Academy  per  controller  costs  found  that 
the  MnATCTC  per-hire  costs  would  be  equal  to  or 
less  than  FAA  Academy  per  controller  costs  by  the 
FYl 998-2000  timeframe. 


The  benefit-to-cost  analysis  found  that  the 
MnATCTC  program  would  begin  returning  a  dollar 
or  more  for  every  dollar  invested  in  the  program  by 
the  FYl 998-2000  timeframe,  given  current  hiring 
projections,  even  with  continued  FAA  financial  sup¬ 
port  of  about  $1.5  million  per  year  to  produce  100 
new  graduates  each  year. 

Capacity  relative  to  projected  demand 

The  MnATCTC  has  a  maximum  capacity,  accord¬ 
ing  to  Congressional  testimony  by  the  program  direc¬ 
tor,  of  producing  100  graduates  per  year  with  a 
continuing  financial  subsidy  from  the  FAA.  Analysis 
of  current  ATCS  workforce  demographics  suggests  a 
much  larger  hiring  demand  after  the  turn  of  the 
century.  It  is  unlikely,  therefore,  that  the  MnATCTC 
can  supply  more  than  a  small  fraction  of  FAA 
workforce  requirements. 

Conclusions 

Overall,  the  MnATCTC  program  appears  to  be 
meeting  most  of  its  stated  objectives.  The  MnATCTC 
is  contributing  to  agency  diversity  goals  for  women, 
but  not  minorities.  The  progress  of  MnATCTC 
graduates  through  field  training  is  essentially  no 
different  than  the  progress  of  FAA  Academy  gradu¬ 
ates.  MnATCTC  were  rated  significantly  lower  on 
five  job  performance  dimensions  than  FAA  Academy 
graduates.  MnATCTC  appears  to  be  able  to  produce 
new  controllers  on  a  cost-competitive  basis  with  the 
FAA  Academy.  FAA  benefits  accruing  largely  from 
avoided  FAA  Academy  training  costs  are  not  likely  to 
balance  cumulative  program  costs  in  the  near-term 
(FYl  997-2000).  Over  the  longer  term,  however,  accrued 
FAA  benefits  maybe  greater  than  cumulative  costs,  even 
with  continued  FAA  financial  support.  Alternative  ap¬ 
proaches  that  require  little  or  no  agency  investment  in 
and  support  of  external  training  infrastructure  at  the 
cost  of  internal  training  capacity  may  yield  greater  rates- 
of-return,  as  well  as  satisfy  larger  proportions  of  the 
agency’s  future  workforce  requirements. 
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SUMMATIVE  EvAUJATION  OF  THE  COLLEGIATE  TRAINING  INITIATIVE 
FOR  Air  Traffic  Control  Speciausts  Program: 
Progress  of  Minnesota  Air  Traffic  Control  Training  Center 
Graduates  in  En  Route  Field  Training 


introduction 

Comprehensive  reviews  of  the  Federal  Aviation 
Administration  (FAA)  air  traffic  control  specialist 
(ATCS)  recruiting,  selection,  and  training  programs 
completed  in  the  late  1980s  (Schultz  &  Marshall- 
Mies,  1988;  Means  et  ah,  1988;  Northern  NEF,  Inc., 
1988)  recommended  that  the  agency  rely  more  heavily 
on  the  technical  training  expertise  available  through 
two-  and  four-year  colleges  and  universities.  In  re¬ 
sponse  to  those  studies,  the  FAA  developed  the  Flight 
Plan  for  Training  {Flight  Plan\  Office  of  Training  and 
Higher  Education,  1989)  in  which  the  development 
of  pre-hire  controller  training  at  the  college  and/or 
university  level  was  identified  as  a  specific  initiative 
(p.  28): 

Pre-hire  training  at  the  college  and  university  level 
will  provide  an  economical  new  source  of  highly  quali¬ 
fied  and  motivated  Air  Traffic  Control  Specialists,  To 
test  this  concept,  a  trial  Air  Traffic  Control  (ATC) 
training  program  will  be  conducted.  A  university-based 
pilot  program  in  which  one  hundred  controllers  will 
earn  undergraduate  degrees  and  receive  the  equivalent 
of  the  FAA  Academy’s  developmental  training  will  be 
initiated  in  1989.  If  hired  by  the  agency,  these  control¬ 
lers  will  enter  the  FAA  training  system  at  an  advanced 
level.  The  agency  will  expand  this  program  to  other 
universities  and  colleges  if  the  experimental  program  is 
successful. 

Program  implementation 

Implementation  of  this  concept  began  in  1989, 
with  approval  and  funding  for  two  programs:  the 
Minnesota  Air  Traffic  Control  Training  Center 
(MnATCTC),  as  administered  by  the  Mid- America 
Aviation  Resource  Consortium,  Eden  Prairie,  Min¬ 
nesota;  and  Hampton  University,  Hampton,  Vir¬ 
ginia.  The  program  was  subsequently  expanded  under 
the  direction  of  the  Higher  Education  and  Advanced 
Technology  Staff  (AHT-30),  culminating  in  the  1991 


FAA  Order  3120.26.  The  order  formally  established 
the  Collegiate  Initiative  for  Air  Traffic  Control  Spe¬ 
cialists  (CTI/ATCS)  as  a  test  program  and  provided 
for  the  selection  of  participating  educational  institu¬ 
tions.  By  1992,  five  institutions  were  participating  in 
the  CTI/ATCS  program: 

•  The  Minnesota  Air  Traffic  Control  Training  Cen¬ 
ter  (MnATCTC),  Eden  Prairie,  Minnesota,  as 
administered  by  the  Mid-America  Aviation  Re¬ 
source  Consortium  (MARC); 

•Hampton  University  (HU),  Hampton,  Virginia; 

•  Community  College  of  Beaver  County  (CCBC), 
Monaca,  Pennsylvania; 

•  University  ofNorth  Dakota  (UND),  Grand  Forks, 
North  Dakota;  and 

•University  of  Alaska  at  Anchorage  (UAA),  An¬ 
chorage,  Alaska. 

However,  full  implementation  of  the  program 
coincided  with  a  significant  downturn  in  ATCS 
hiring  requirements  in  1992  and  1993  (Morrison, 
Fotohui,  &  Broach,  1996),  resulting  in  a  lower  rate  of 
hiring  of  CTI/ATCS  program  graduates  than  had 
been  expected.  The  Office  of  the  Inspector  General 
for  the  Department  of  Transportation  (DOT/IG) 
conducted  an  audit  of  FAA  higher  education  pro¬ 
grams  at  about  the  same  time.  That  DOT/IG  (1993, 
p.  14-15)  recommended  that  the  FAA  discontinue 
future  funding  for  the  CTI/ATCS  program  and  ad¬ 
vise  the  appropriate  congressional  committees  of  the 
program’s  limited  success,  absence  of  hiring  oppor¬ 
tunities  for  program  graduates,  and  intention  to 
recommend  discontinuance  of  the  program.  How¬ 
ever,  the  FAA  formally  disagreed  with  the  DOT/IG 
recommendations  on  the  CTI/ATCS  program,  pend¬ 
ing  an  evaluation  of  the  long-range  recruitment  need 
for  air  traffic  control  specialists.  Moreover,  at  the 
time  of  the  DOT/IG  audit,  only  a  few  (61)  CTI/ 
ATCS  program  graduates  had  been  hired  by  the  FAA 
from  the  MnATCTC  program.  Since  that  date,  the 
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number  of  graduates  from  MnATCTC  and  other 
participating  institutions  hired  by  the  FAA  has  in¬ 
creased  substantially  (Table  1),  providing  a  larpr 
sample  to  support  an  empirically  based  summative 
evaluation.  As  shown  in  Table  1,  the  largest  number 
of  CTI/ATCS  graduates  hired  by  the  FAA  has  come 
from  the  MnATCTC  program.  In  addition,  the 
MnATCTC  program  has  been  a  subject  in  budget 
negotiations  between  the  Congress  and  the  FAA  since 
FY94.  Therefore,  this  report  focuses  specifically  on 
the  MnATCTC  program  in  an  empirical  evaluation 
of  goal  attainment  relative  to  agency  hiring  require¬ 
ments,  costs,  and  benefits. 

Evaluation  design 

Evaluations  can  be  broadly  categorized  into  three 
classes,  based  on  the  primary  focus  of  the  research:  (a) 
program  conceptualization;  (b)  program  implementa¬ 
tion;  and  (c)  program  utility  (Rossi  &  Freeman,  1985). 
The  formative  evaluation  conducted  in  fiscal  year  1993 
(Morrison,  Fotohui,  &  Broach,  1996)  concentrated  on 
program  implementation  at  each  of  the  five  participat¬ 
ing  educational  institutions.  That  evaluation  specifi¬ 
cally  addressed  the  degree  of  innovation  that  the  five 
participating  institutions  demonstrated  with  regard  to 
their  (a)  recruiting  activities,  (b)  selection  procedures, 
and  (c)  training  methods.  Overall,  the  participating 
institutions,  including  MnATCTC,  had  developed  re¬ 
cruiting,  selection,  and  training  methodologies  and 
technologies  that  differed  substantially  from  those  used 
by  the  FAA.  This  summative  evaluation  addresses  out¬ 


comes,  with  particular  attention  to  the  MnATCTC 
program.  MnATCTC  graduate  diversity  and  perfor¬ 
mance  in  field  facility  training  is  compared  with  that  of 
a  control  group  of  FAA  Academy  graduates. 

It  should  be  noted  that  the  control  group  in  this 
evaluation  is  not  strictly  equivalent  to  the  “experi¬ 
mental”  group  of  MnATCTC  graduates.  For  ex¬ 
ample,  the  control  group  is  not  drawn  from  the  same 
en  route  facilities  to  which  MnATCTC  graduates 
were  assigned  (Table  2).  A  third  of  the  MnATCTC 
graduates  were  assigned  to  the  Minneapolis  Air  Route 
Traffic  Control  Center  (ARTCC);  another  20%  were 
assigned  to  the  Oakland  ARTCC.  In  contrast,  the 
control  group  of  FAA  Academy  graduates  were  scat¬ 
tered  across  20  of  23  en  route  centers  (excluding 
Guam).  Nor  is  the  control  group  matched  to  the 
MnATCTC  graduates  in  terms  of  hiring  dates  (Table 
3).  The  control  group  entered  field  training  in  mid- 
1991,  while  the  MnATCTC  graduates  entered  ser¬ 
vice  from  late  1991  through  1993.  As  a  consequence, 
the  control  group  had  been  at  the  first  assigned  en 
route  facility  an  average  of  38  months  {SD  =11.3 
months),  compared  with  28  months  (^SD  =  9.32 
months)  for  the  MnATCTC  graduates  {F  (1,291)  = 
71.80,  p  <  .001),  as  of  June  1995.  Therefore,  the 
research  design  can  be  characterized  as  a  post-treat¬ 
ment  comparison  with  a  non-equivalent  control 
group,  which  is  considered  a  relatively  “weak”  design 
in  terms  of  controls  for  validity  and  generalizability 
(Campbell  &  Cook,  1976).  However,  this  design 
does  provide  at  least  some  objective  data  as  the  basis 


Table  1.  CTI/ATCS  Hiring  by  fiscal  year  and  participating  institution® 


Participating  Institution _ _ 

MnATCTC  HU  CCBC  UNO _ UAA _ Tote 

3 
36 
21 
96 
53 
80 
29 
318 

Notes:  ®Data  provided  by  Aviation  Careers  Examining  Division  (AMH-300) 

‘‘FY88-90  CCBC  Cooperative  Education  Program  graduates  counted  as 
CTI/ATCS  hires 


88-90^ 

91 

26 

3 

10 

92 

19 

2 

93 

78 

18 

94 

40 

1 

0 

95 

30 

14 

35 

96 

17 

6 

Total 

193 

32 

74 

5 
1 

6 
12 


2 


Table  2.  Air  Route  Traffic  Control  Center  (ARTCC)  assignments  by  program 


FACID 

Name 

MnATCTC 

FAA  Academy 

N 

% 

N 

% 

ZAB 

Albuquerque 

9 

5.7 

ZAN 

Anchorage 

7 

5.1 

ZAU 

Chicago 

10 

6.4 

ZBW 

Boston 

2 

1.3 

ZDC 

Washington 

1 

0.7 

8 

5.1 

ZDV 

Denver 

11 

8.1 

ZFW 

Fort  Worth 

5 

3.7 

12 

7.6 

ZHN 

Honolulu 

2 

1.3 

ZHU 

Houston 

11 

7.0 

ZID 

Indianapolis 

5 

3.7 

7 

4.5 

ZJX 

Jacksonville 

1 

0.7 

8 

5.1 

ZKC 

Kansas  City 

18 

13.2 

ZLA 

Los  Angeles 

2 

1.9 

3 

1.5 

ZLC 

Salt  Lake  City 

1 

0.7 

7 

4.5 

ZMA 

Miami 

15 

9.6 

ZME 

Memphis 

3 

2.2 

16 

10.2 

ZMP 

Minneapolis 

46 

33.8 

11 

7.0 

ZNY 

New  York  City 

2 

1.5 

12 

7.6 

ZOA 

Oakland 

1 

0.6 

27 

19.9 

ZOB 

Cleveland 

11 

7.0 

ZSE 

Seattle 

9 

5.7 

ZSU 

San  Juan  CERAP 

1 

0.7 

ZTL 

Atlanta 

6 

4.4 

3 

1.9 

for  assessing  MnATCTC  program  goal  attainment, 
costs,  and  benefits  relative  to  the  known  standard  of 
FAA  Academy  graduates. 

Goals,  costs,  and  benefits  should  be  operationally 
defined  in  evaluation  research  such  as  this  study 
(Bloom,  1967;  Rossi  &  Freeman,  1985).  Specific 
operationally-defined  criteria  for  the  summative  evalu¬ 
ation  of  the  MnATCTC  program  relative  to  tradi¬ 
tionally  hired  and  trained  FAA  Academy  graduates 
included  (a)  increased  employee  diversity,  (b)  im¬ 
proved  employee  performance  as  represented  by  in¬ 
structor,  self,  and  supervisor  evaluations,  (c)  reduction 
in  time  for  graduates  of  these  programs  to  complete 
the  ATCS  field  training  sequence,  and  (d)  reduction 
of  attrition  during  field  training  among  MnATCTC 
graduates.  The  latter  two  operational  criteria,  along 
with  specific  incurred  and  avoided  costs  associated 
with  the  program,  provide  a  basis  for  assessing  the 
relative  benefits  of  the  initiative. 


METHOD 

Sample 

Two  groups  of  controllers  were  identified  as  the 
sample  for  this  evaluation:  136  MnATCTC  gradu¬ 
ates  hired  by  the  FAA  between  1991  and  1993  (see 
Table  3  for  hiring  dates),  and  the  157  FAA  Academy 
graduates  (“FAA  Academy”)  that  entered  en  route 
ATCS  field  training  in  May  and  June  1991.  The  FAA 
Academy  graduates  included  persons  who  had  par¬ 
ticipated  in  the  development  and  validation  of  the 
FAA’s  ATCS  Pre-Training  Screen  (Broach  &  Brecht- 
Clark,  1994).  Facility  assignments  and  hiring  dates 
by  program  are  presented  in  Tables  2  and  3.  The  overall 
demographic  characteristics  by  program  are  described 
in  Tables  4  (minority  status  and  gender)  and  5  (Equal 
Employment  Opportunity  (EEO)  subgroups). 
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Table  3.  Facility  reporting  dates  by  program 


Reporting  Pate 
May  1991 
June  1991 
September  1991 
November  1991 
March  1992 
April  1992 
May  1992 
November  1992 
January  1993 
February  1993 
March  1993 
April  1993 
May  1993 
June  1993 
July  1993 
October  1993 
November  1993 
December  1 993 
January  1994 


MnATCTC 


N 


% 


FAA  Academy 


N 


% 


72 

45.9 

83 

52.9 

15 

11.0 

2 

1.2 

11 

8.1 

16 

11.8 

1 

0.7 

3 

2.2 

2 

1.4 

27 

19.9 

3 

2.2 

11 

8.1 

2 

1.5 

12 

8.8 

11 

8.0 

2 

1.5 

1 

0.7 

10 

7.3 

6 

4.4 

3 

2.2 

Table  4.  Minority  status  and  gender  of  evaluation  sample 


Characteristic 

MnATCTC 

FAA  Academy 

Total 

N 

% 

N  % 

N 

% 

Minority  status 

Nonminority 

111 

81.4% 

136  86.6% 

247 

84.3% 

Minority 

12 

8.8% 

21  13.4% 

33 

1 1 .3% 

Missing 

13 

3.7% 

13 

4.4% 

Gender 

Male 

82 

60.3% 

130  82.8% 

212 

72.4% 

Female 

54 

39.7% 

27  17.2% 

81 

27.6% 

Missing 

Measures 

Dependent  variables,  representing  the  criteria  for 
this  summative  evaluation,  included  (a)  representa¬ 
tion  by  EEO  group,  (b)  rate  of  progress  in  field 
training,  (c)  attrition  from  en  route  field  training,  (d) 
on-the-job  training  instructor  (OJTI),  self,  and  su¬ 
pervisor  ratings  of  field  performance,  and  (e) 
MnATCTC  program  costs  and  benefits. 


Training  progress 

One  means  of  achieving  the  goal  of  safe,  orderly, 
and  efficient  operation  of  the  National  Airspace 
System  is  to  ensure  that  the  persons  directing  air 
traffic  have  reached  the  full  performance  level  (FPL). 
FPL  controllers  are  the  backbone  of  the  controller 
workforce,  as  they  are  responsible  on  a  moment-to- 
moment  basis  for  the  safe,  orderly,  and  expeditious 
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flow  of  air  traffic  through  the  National  Airspace 
System.  Development  of  FPL  skills  and  knowledge 
requires  extended,  intensive,  and  expensive  formal 
and  on-the-job  training  for  trainee  (“developmen¬ 
tal”)  controllers.  The  training  averages  three  years  in 
en  route  centers  (Manning,  Della  Rocco,  &  Bryant, 
1989).  This  extended  field  training  represents  a  sig¬ 
nificant  agency  expense  directly  proportionate  to  the 
time  spent  in  training.  For  example,  the  Air  Traffic 
Training  Work  Group  (ATTWG;  1991a)  in  a  com¬ 
prehensive  review  of  ATCS  training  programs,  esti¬ 
mated  field  training  costs  at  about  $131,739  per  en 
route  controller  over  a  36-month  interval.  Innovative 
programs  such  as  those  offered  at  MnATCTC  have 
the  potential  to  reduce  overall  training  times,  thereby 
reducing  the  agency’s  costs.  Similarly,  attrition  from 
the  field  training  sequence  represents  a  lost  invest¬ 
ment  for  the  agency.  Reductions  in  attrition  rates, 
therefore,  translate  directly  into  cost  savings. 

Performance  ratings 

Evaluations  of  technical  skill,  knowledge,  and 
teamwork  provide  additional  information  about  core 
job  performance  not  available  in  the  training  progress 
or  attrition  measures.  Core  technical  job  perfor¬ 
mance  may  be  thought  of  as  the  product  of  knowl¬ 
edge,  skill,  and  motivation  (Campbell,  1990; 
Campbell,  McCloy,  Oppler,  &  Sager,  1993).  Items 
to  assess  individual  technical  knowledge  and  skill 
were  adapted  directly  from  the  existing  instruments 
used  for  over-the-shoulder  evaluation  of  technical 
performance  (FAA  Form  3120-25).  Teamwork  is  a 
dimension  of  controller  training  and  job  perfor¬ 
mance  that  has  become  a  recent  focus  of  concern  (Air 
Traffic  Training  Workgroup,  1991a,b;  E.  L.  Hamm 
&  Associates,  Inc.,  &  Hampton  University,  1990; 


Hartel  &  Hartel,  1995;  Newcomb  &  Jerome,  1994; 
Seamster,  Cannon,  Pierce,  &  Redding,  1992;  Sherman 
&  Helmreich,  1993).  For  example,  teamwork  was 
incorporated  as  an  explicit  evaluation  dimension  in 
the  post-training  performance  verification  process 
(Performance  Verification  Division,  undated).  Items 
representing  teamwork  in  the  evaluation  instrument 
were  based  on  a  review  of  the  teamwork  and  interper¬ 
sonal  skills  literature  with  reference  to  air  traffic 
control  job  analysis  information  (Stark,  1994). 

In  addition,  concerns  were  raised  by  participating 
institutions  that  the  degree  to  which  a  CTI  graduate 
was  accepted  in  the  facility  by  co-workers,  instruc¬ 
tors,  and  management  might  influence  OJT  assign¬ 
ments,  training,  and  perceptions  of  performance. 
Items  were  included,  therefore,  in  the  performance 
rating,  to  represent  the  degree  to  which  the  ratee  felt 
or  was  perceived  as  being  accepted  in  the  workplace. 
Finally,  a  global  subjective  assessment  of  the  ratee’s 
potential  to  succeed  in  the  ATCS  occupation  was 
incorporated  into  the  performance  rating  instrument. 
Three  parallel  versions  of  the  instrument  were  devel¬ 
oped  for  first  line  supervisors,  OJT  instructors,  and 
incumbent  controllers.  A  sample  of  the  OJT  instruc¬ 
tor,  controller,  and  supervisor  evaluation  forms  and 
associated  cover  letters  are  provided  in  Appendix  A. 

Benefits  and  costs 

The  training  and  survey  data  described  above 
provide  information  about  the  degree  to  which  the 
MnATCTC  is  meeting  its  program  objectives,  in 
terms  of  the  progress  and  performance  of  program 
graduates,  in  comparison  with  controllers  entering 
the  occupation  through  the  FAA  Academy  program. 
While  knowledge,  both  of  the  manner  in  which  a 
program  such  as  the  Minnesota  program  has  been 


Table  5.  Representation  of  equal  employment  opportunity  (EEO)  groups 


Characteristic 

MnATCTC 

FAA  Academy 

Total 

N 

% 

N 

% 

N 

% 

Nonminority  Male 

66 

48.5% 

111 

70.7% 

177 

60.4% 

Nonminority 

45 

33.1% 

25 

15.9% 

70 

23.9% 

Female 

Minority  Male 

9 

6.6% 

19 

12.1% 

28 

9.6^ 

Minority  Female 

3 

2.2% 

2 

1.3% 

5 

1 .7% 

Missing 

13 

9.6% 

13 

4.4% 

5 


implemented,  and  of  its  outcomes,  is  indispensable 
to  program  managers,  stakeholders,  and  policymakers, 
it  is  just  as  critical  in  the  evaluation  process  to  provide 
information  about  costs  relative  to  benefits  (Rossi  & 
Freeman,  1985).  Inputs  to  the  program  and  out¬ 
comes  are  measured  in  monetary  terms  in  cost-ben¬ 
efit  analyses  to  support  program  evaluation.  Therefore, 
the  final  step  in  the  evaluation  was  to  develop  a 
framework  for  measuring  the  costs  associated  with 
and  benefits  accruing  from  the  CTI/ATCS  program. 
That  framework  was  then  applied  to  the  evaluation  of 
the  MnATCTC  program  as  a  model  for  future  cost- 
benefit  evaluations  for  each  institution  participating 
in  the  overall  CTI/ATCS  program. 

Procedure 

Field  training  progress  data  collection 

Field  training  data  for  both  samples  were  extracted 
from  the  Civil  Aeromedical  Institute  ATCS  Training 
Tracking  (TRACKING)  database.  This  database  was 
maintained  under  the  FAA  ATCS  National  Training 
Tracking  Program  order  (FAA  Order  3120.22A;  FAA, 
1985)  through  June  1995.  The  phases  of  training  are 
described  in  Table  6,  based  on  the  1988  En  Route 
Instructional  Program  Guide  (IPG;  FAA,  1988). 
There  are  variations  allowed  in  the  training  phases, 
based  on  local  facility  requirements,  as  noted  in 
Table  6.  For  example.  Phase  VII  (Preliminary  Ra¬ 
dar-Associated/Nonradar  Control  Training  and 
Assistant  Controller  Duties)  is  conducted  in  the 
classroom  for  up  to  8  weeks  (40  days).  The  variation 
named  Phase  VIIB  adds  15  nonradar  familiarization 
problems  in  the  facility  dynamic  simulation  labora¬ 
tory  (“DYSIM”)  to  the  classroom  instruction.  Other 
phases  with  variations  include  Phase  VIII  (3  varia¬ 
tions,  differing  in  the  total  number  and  mix  of 
nonradar  and  radar  problems  run  in  the  DYSIM), 
Phase  XI  (2  variations,  differing  in  the  total  number 
of  hours  and  DYSIM  radar  problems),  and  Phase 
XIII  (2  variations,  differing  in  the  total  number  of 
OJT  hours  allowed  per  sector).  The  sequences  of 
phases  and  their  variations  (known  as  a  “tracks”  or 
curricula)  taken  by  each  controller,  are  summarized 
in  Table  7.  The  majority  of  both  MnATCTC  and 
FAA  Academy  graduates  followed  either  what  is 
known  as  the  “B-track”  (substituting  phases  VIIB, 
VIIIB,  XIB,  and  XIIIB  for  the  standard  phases  VII, 
VIII,  XI,  and  XIII),  or  what  is  called  the  “AB  Stan¬ 
dard”  track,  with  phases  VIIIA  and  XIB  in  place  of 


the  standard  phase  VIII  and  XI.  A  majority  of  the 
MnATCTC  (53.7%)  graduates  followed  the  AB- 
Standard  track,  with  a  substantial  minority  (23.5%) 
following  the  B-track.  In  comparison,  a  plurality  of 
FAA  Academy  graduates  (42.0%)  pursued  the  B- 
Track,  with  another  42.7%  following  the  AB-Stan- 
dard  curriculum.  Overall,  despite  the  differences  in 
facility  assignments,  the  curricular  tracks  followed  by 
graduates  from  the  two  programs  (Academy, 
MnATCTC)  are  reasonably  similar. 

Training  data  available  included  the  number  of 
days  spent  in  each  phase  of  training,  hours  of  on-the- 
job  training  (OJT)  taken  in  each  phase,  and  the 
overall  rating  of  performance  in  that  phase  on  a  1 
(lowest  10%  of  controllers  observed)  to  6  (top  10% 
of  all  controllers  observed)  scale.  The  availability  of 
data  for  subjects  in  each  phase  of  training  was  entirely 
dependent  upon  the  timeliness  of  reporting  facilities; 
all  data,  as  reported  by  the  facilities,  were  extracted 
for  this  analysis.  The  reliability  of  the  ATCS  training 
tracking  data  has  been  described  by  Manning  (1990). 
Incoming  data  were  closely  examined  by  CAMI  re¬ 
search  technicians  for  out-of-range  and  missing  val¬ 
ues;  follow-up  calls  were  made  to  the  reporting  facility 
as  required  to  verify  and  complete  data.  The  time- 
based  measures  (days  in  phase;  hours  of  OJT)  are 
ratio  scale  variables;  any  unreliability  in  those  mea¬ 
sures  would  be  due  to  errors  in  reporting  at  the 
facility  level,  or  to  clerical  errors  in  data  handling. 
The  reliability  of  the  subjective  assessment  of  devel¬ 
opmental  performance  in  a  phase  of  training  (the  IP) 
is  undocumented.  However,  studies  of  performance 
ratings  in  the  literature  suggest  that  such  ratings  are 
reasonably  reliable  and  useful  for  avariety  orworkforce 
research  purposes  (Borman,  M7hite,  Pulakos,  & 
Oppler,  1991;  Hoffman,  Nathan,  &  Holden,  1991; 
Smith,  1976). 

Attrition  data  collection 

Data  to  identify  attritions  from  training  for  both 
samples  were  extracted  from  the  FAA  Consolidated 
Personnel  Management  Information  System  (CPMIS) 
and  cross-referenced  with  the  CAMI  ATCS  Training 
Tracking  (TRACKING)  database.  There  is  no  single 
field  in  either  data  source  indicating  a  training  attri¬ 
tion;  rather,  multiple  data  elements  from  both  sources 
must  be  evaluated  and  combined  to  determine  the 
outcome  for  a  given  case.  Outcome  coding  for  this 
analysis  was  based  on  CPMIS  and  TRACKING  data 
fields  representing  (a)  grade  level,  (b)  training  phase 
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Table  6.  En  route  training  curriculum 
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Table  7.  Enroute  OJT  curriculum  track  followed  by  group 


MnATCTC 

FAA  Academy 

Track 

Phases* 

N 

% 

N  % 

Standard 

V,  VI,  VII,  VIII,  IX.  X, 

XI,  XII,  XIII 

7 

5.1 

5  3.2 

A  Track 

V,  VI.  VII.  VIIA,  IX,  X, 

XI.  XII.  XIII 

11 

8.1 

B  Track 

V,  VI,  VIIB,  VIIIB,  IX,  X, 
XIB,  XII,  XIIIB 

32 

23.5 

66  42.0 

8B  Track: 

V,  VI,  VII,  VIIIB,  IX,  X. 

XI,  XII,  XIII 

1 

0.7 

9  5.7 

B  Variation  1 

V,  VI,  VIIB,  VIIIB,  IX,  X, 
XIB,  XII,  XIII 

4 

2.9 

B  Variation  2 

V.  VI,  VII,  VIII.  IX,  X, 

XIB,  XII,  XIII 

1 

0.7 

10  6.4 

AB  Track 

V,  VI.  VII.  VIIIA,  IX,  X, 
XIB,  XII,  XIIIB 

7 

5.1 

Notes:  “Variations  in  phases  for  each  track  shown  in  boldface. 


completions,  (c)  training  phase  grades,  (d)  facility 
type  and  level,  (e)  facility  disposition  codes,  and  (f) 
facility  types  and  levels  at  time  of  entry  into  field 
training  and  at  time  of  data  extraction.  Possible 
training  outcomes  at  the  first  assigned  field  facility 
included:  (1)  separation  from  the  ATCS  occupation 
(which  may  or  may  not  involve  termination  from 
employment  by  the  FAA);  (2)  attrition  (without 
separation  from  the  FG-2152  occupation  or  agency) 
from  the  first  en  route  facility  through  facility  or 
option  change  (e.g.,  switch  to  terminal  or  flight 
service);  (3)  still  in  developmental  (training)  status  as 
a  controller  at  the  first  en  route  facility;  and  (4) 
achieved  Full  Performance  Level  (FPL)  in  the  ATCS 
occupation  as  of  June  1995  at  the  first  en  route 
facility.  Only  outcomes  at  the  first  assigned  facility 
are  considered  in  this  analysis  for  two  reasons. 

First,  the  training  tracking  system  was  originally 
designed  to  follow  progress  through  OJT  to  the  FPL 
at  the  first  facility;  the  data  for  second  or  third 
assignments  are  both  less  reliable  and  less  complete. 
Second,  there  are  significant  financial  costs  associ¬ 
ated  with  moves  from  the  first  assigned  facility  prior 
to  achievement  of  the  FPL  certification.  Those  costs 
may  include,  but  are  not  limited  to,  expended  train¬ 


ing  funds,  time,  permanent  change  of  station  (e.g., 
moving)  costs,  and  personnel  replacement  costs. 
Therefore,  improved  outcomes  at  the  first  facility  can 
lead  to  significant  avoided  costs  for  the  FAA. 

Performance  ratings  data  collection 

Survey  administration.  Facilities  to  which 
MnATCTC  and  FAA  Academy  graduates  had  been 
assigned  were  identified  through  CPMIS,  as  shown 
in  Table  2.  Working  through  the  Air  Traffic  chain  of 
command,  supervisors  and  OJT  instructors  of 
MnATCTC  and  FAA  Academy  graduates  were  iden¬ 
tified  by  the  facilities.  A  mail-merge  database,  linking 
controller,  supervisor,  and  OJT  instructor,  was  de¬ 
veloped  on  the  basis  of  the  data  provided  by  facilities 
to  support  mailing  a  ratings  package.  The  ratings 
package  included  a  cover  letter  from  the  Director  of 
Air  Traffic  Program  Management  (ATZ-1),  an  expla¬ 
nation  of  the  project,  instructions  for  rating,  and  the 
rating  form.  The  cover  letter  was  addressed  to  the 
rater  by  name,  using  the  mail-merge  database.  The 
supervisor  and  OJT  instructor  forms  indicated  the 
name  ofthe  incumbent  controller  to  be  rated.  Reminder 
cards  were  mailed  to  controllers,  supervisors,  and  in¬ 
structors  about  one  month  after  the  initial  mailing. 
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Survey  response  rates.  Performance  evaluation 
survey  response  rates  are  presented  in  Table  8.  OJT 
instructors  for  53  (39.0%)  of  the  MnATCTC  gradu¬ 
ates  returned  surveys,  compared  with  61  (38.9%) 
from  OJT  instructors  of  the  FAA  Academy  gradu¬ 
ates.  The  return  rates  for  incumbent  controllers  from 
both  groups  were  better,  with  72.1%  (98)  of  the 
MnATCTC  graduates  returning  self-evaluation  sur¬ 
veys  compared  with  68.1%  (107)  of  the  FAA  Acad¬ 
emy  graduates.  Supervisors  for  64  of  the  136 
MriATCTC  graduates  (47.1%)  and  95  of  157  FAA 
Academy  graduates  (60.5%)  returned  surveys.  The 
instructor  and  incumbent  return  rates  for  each  group 


were  not  statistically  different.  However,  the  return 
rate  for  supervisors  of  the  FAA  Academy  group 
(60.5%)  was  significantly  greater  than  the  return 
rate  for  supervisors  of  the  MnATCTC  group 
(37.2%;  Z=  -3.25, /.<  .001). 

Respondent  characteristics.  Demographic  char¬ 
acteristics  of  the  controllers  that  returned  perfor¬ 
mance  evaluation  surveys  are  presented  in  Table  9. 
The  majority  of  the  MnATCTC  graduates  returning 
surveys  were  male  (72.0%),  as  were  the  FAA  Academy 
graduates  (68 . 5%) .  All  1 2  of  the  minority  MnATCTC 
graduates,  and  20  of  2 1  minority  FAA  Academy  gradu¬ 
ates  returned  performance  self-evaluation  surveys. 


Table  8.  Performance  evaluation  survey  return  rates 


MnATCTC 

FAA  Academy 

OJT  Instructors 

Mailed 

136 

157 

Total  returns 

53 

61 

Return  rate 

39.0% 

38.9% 

Controllers 

Mailed  Controllers 

136 

157 

Total  returns 

98 

107 

Return  rate 

72.1% 

68.1% 

Supervisors 

Mailed  Supervisors 

136 

157 

Total  returns 

64 

95 

Return  rate 

47.1% 

60.5%*** 

***p<.001 


Table  9.  Demographic  characteristics  of  controller  survey  respondents 


Group 


MnATCTC _ 

^Group  ^respond  'y^Respond 


Academy 

^Group  ^Respond  **^^Respond 


Gender 


Male 

82 

59 

72.0% 

130 

Female 

54 

39 

72.2% 

27 

Missing 

Minority 

'  Status 

Non-minority 

111 

77 

69.4% 

136 

Minority 

12 

12 

100.0% 

21 

Missing 

13 

9 

69.2% 

89 

18 


68.5% 

66.7% 


11 


87 

20 


64.0% 

95.2% 


OJT  instructors  returning  surveys  had  been  con¬ 
trollers  for  an  average  of  13  years  (SD  =  6.7  years). 
These  instructors  had  been  at  their  current  facility 
about  10  years  {SD  =  5.3  years)  and  had  been  in  their 
current  position  as  an  instructor  about  8  years  on  the 
average  {SD  =  5.2  years).  They  had  been  providing 
training  to  the  developmental  being  rated  an  average 
of  12.9  months  {SD  =  24.4  months).  There  were  no 
statistically  significant  differences  on  these  measures 
between  OJT  instructors  for  MnATCTC  and  FAA 
Academy  graduates. 

The  supervisors  who  provided  performance  evalu¬ 
ation  ratings  on  MnATCTC  and  FAA  Academy 
graduates  for  research  purposes  only  in  this  study 
averaged  18.3  years  {SD  =  6.7)  of  experience  in  the 
ATCS  occupation,  with  average  of  12.5  years  {SD  = 
7.5)  in  the  current  facility.  These  supervisors  had  an 
average  of  5.2  years  {SD  =  3.8)  of  experience  in  their 
current  positions  as  supervisors  and  had  been  super¬ 
vising  the  rated  CTI/ATCS  or  Academy  graduate  an 
average  of  14.2  months  {SD  =  10.2).  There  were  no 
significant  differences  between  supervisors  of 
MnATCTC  and  FAA  Academy  graduates  on  the 
times  as  controller,  supervisor,  or  at  the  current 
facility.  However,  supervisors  of  Academy  graduates 
reported  significantly  longer  times  supervising  the 
rated  incumbent  {M=  15.9,  SD  =  10.7  months)  than 
supervisors  of  MnATCTC  graduates  (M=  10.7,  SD 
=  8.0  months;  /(j  ,,5)  =  3.36,  p  <  .001).  This  is 
consistent  with  the  previous  finding  that  FAA  Acad¬ 
emy  graduates  had  been  at  the  first  facility  longer 
than  the  MnATCTC  graduates. 

Costs  and  benefits 

Framework.  Thompson  (1980)  identified  the  fol¬ 
lowing  major  steps  in  cost-benefit  analysis  for  pro¬ 
gram  evaluation:  (1)  identify  the  decision-maker(s) 
who  will  use  the  results  of  the  evaluation;  (2)  identify 
alternatives;  (3)  identify  costs;  (4)  identify  benefits; 
(5)  value  program  effects  in  dollars;  (6)  discount 
those  values  for  the  effects  of  time;  (7)  take  distribu¬ 
tional  effects  into  account,  as  appropriate;  and  (8) 
aggregate  and  interpret  the  valued  effects.  This  pro¬ 
gram  evaluation  is  intended  to  serve  the  decision¬ 
making  requirements  of  FAA  Air  Traffic  Services 
management.  There  are  two  alternatives  considered 
in  the  evaluation;  training  entry-level  controllers  at 
(a)  MnATCTC  or  (b)  the  FAA  Academy.  Costs, 
benefits,  and  the  valuation  of  program  effects  in 
dollars  are  discussed  below  in  detail.  Opportunity 


costs,  time-related  discounts  in  the  value  of  money, 
and  distributional  effects  were  not  addressed  in  this 
initial  cost-benefit  analysis. 

Costs.  The  agency  incurs  both  direct  and  indirect 
costs  in  administering  the  MnATCTC  and  other 
programs  under  the  CTI/ATCS  charter.  Direct  costs 
to  the  agency  include  the  Congressionally-mandated 
funds  invested  in  the  MnATCTC.  Direct  costs  were 
obtained  on  the  basis  of  public  laws  passed  by  the 
U.S.  Congress  (see  Morrison,  Fotohui,  &  Broach, 
1996,  for  citations).  Indirect  costs  include  (a)  pro¬ 
gram  management  staff  time,  (b)  evaluation  costs, 
and  (c)  other  staff  costs  associated  with  supporting 
the  programs.  Indirect  costs  were  estimated  on  the 
basis  of  an  electronic  mail  survey  in  late  1994  of  FAA 
program  offices  and  managers.  The  survey  requested 
estimates  of  the  proportion  of  staff  time  and  travel 
spent  on  the  program  by  fiscal  year.  These  indirect 
costs  were  then  projected  forward  to  provide  esti¬ 
mates  through  the  year  2003. 

Benefits.  Four  major  classes  of  direct,  quantifiable 
benefits  that  could  be  expressed  in  terms  of  dollars 
were  identified  for  the  cost-benefit  analysis.  First, 
under  the  program  structure  described  in  the  forma¬ 
tive  evaluation  report  (Morrison,  Fotohui,  &  Broach, 
1996),  MnATCTC  graduates  took  the  written  ATCS 
aptitude  test  battery  but  bypassed  the  screening  pro¬ 
gram  at  the  FAA  Academy  that  was  in  place  through 
March  of  1992.  MnATCTC  graduates  also  bypassed 
the  successor  five-day  computerized  assessment  of 
aptitude  that  was  implemented  in  June  1992  to 
replace  the  former  nine-week  FAA  Academy  ATCS 
Nonradar  Screen  (Broach  &  Brecht-Clark,  1994). 
Therefore,  the  first  benefit  was  avoided  screening 
costs  for  Mn AT CT C  graduates.  Second ,  MnAT CT C 
graduates  bypassed  the  FAA  Academy  en  route  train¬ 
ing  program  as  implemented  in  June  1992,  and  were 
placed  directly  into  field  training.  Therefore,  the 
second  monetary  benefit  to  the  agency  was  avoided 
training  costs  at  the  FAA  Academy.  Third,  the  perfor¬ 
mance  verification  (PV)  function  was  delegated  to 
the  MnATCTC  from  June  1992  onwards;  therefore, 
the  third  monetary  benefit  to  the  FAA  was  avoided 
costs  for  PV.  Finally,  as  described  in  the  field  training 
measures,  it  was  expected  that  MnATCTC  graduates 
would  take  about  the  same  time,  or  less,  than  FAA 
Academy  graduates  to  attain  certification  as  FPL  en 
route  controllers.  Therefore,  the  fourth  monetary 
benefit  to  the  agency  was  savings  associated  with 
reduced  times  to  FPL  certification  for  MnATCTC 
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graduates.  Published  figures  were  used  to  estimate 
costs  of  screening,  FAA  Academy  training,  and  field 
training  savings.  Cumulative  savings  were  projected 
forwards  through  the  year  2003,  as  with  costs. 

Analyses 

Employee  diversity 

The  representation  of  women  and  minorities  in 
the  MnATCTC  sample  was  compared  with  their 
representation  in  the  FAA  Academy  sample.  Fisher’s 
Z-test  was  used  to  compare  the  proportions  ofwomen 
and  minorities  in  the  two  evaluation  samples,  on  the 
hypothesis  that: 

Hji  Women  and  minorities  will  be  equally  repre¬ 
sented  in  the  MnA  TCTC  and  FAA  Academy  samples. 

Field  training  progress 

The  average  number  of  days,  hours  of  OJT,  and 
ratings  of  performance  in  each  phase  of  the  field 
training  curriculum  were  compared  through  a  one¬ 
way  analysis  of  variance  between  the  two  evaluation 
samples.  However,  results  based  on  the  raw  numbers 
of  days  in  phase,  hours  of  OJT,  and  indicators  of 
performance  can  be  misleading  due  to  differences  in 
training  programs  between  air  route  traffic  control 
centers  (ARTCCs),  as  well  as  curriculum  differences 
noted  in  Table  7.  For  example,  the  historical  average 
time  to  complete  the  same  phase  of  en  route  training 
at  one  ARTCC  may  be  very  different  from  times  at 
other  centers.  These  differences  may  be  attributable 
to  variables  such  as  traffic  patterns,  facility  resources, 
and  training  loads  (General  Accounting  Office, 
1989a).  As  a  result,  the  variability  in  the  raw  numbers 
of  days  in  phase,  hours  of  OJT,  and  indicators  of 
performance  may  be  more  attributable  to  inter-facil¬ 
ity  differences  rather  than  to  between-group  differ¬ 
ences.  To  compensate,  in  some  degree,  for  the 
influence  of  in  ter- facility  differences  on  average  times 
in  training,  the  training  measures  for  the  evaluation 
samples  were  standardized  relative  to  the  historical 
means  and  standard  deviations  for  each  center  to 
which  subjects  were  assigned.  The  analysis  of  vari¬ 
ance  was  then  performed  on  these  standardized  train¬ 
ing  progress  measures.  The  working  hypothesis  for 
analysis  of  the  training  measures  was: 

The  progress  of  MnA  TCTC  graduates  through 
field  training  will  be  the  same  as  FAA  Academy  gradu¬ 
ates,  as  indexed  by  standardized  days  spent,  hours  of 
OJT  taken,  and  indicators  of  performance  earned  in 
each  phase  of  en  route  field  training. 


Attrition 

Continued  employment  in  the  ATCS  occupation, 
according  to  FAA  Order  3330. 30C  (FAA,  1984),  is 
contingent  upon  satisfactory  progression  to  the  full 
performance  level.  Failure  to  progress  in  training  may 
be  the  basis  for  separation  from  the  GS-21 52  occupa¬ 
tion.  Alternatively,  an  individual  not  progressing 
satisfactorily  in  field  training,  as  described  under 
FAA  Order  3120. 24A  (FAA,  1993),  might  be  re¬ 
tained  in  the  ATCS  occupation,  if,  and  only  if,  he  or 
she  has  “shown  potential  for  work  at  the  full  perfor¬ 
mance  level  in  different  facilities”  (FAA,  1984,  p.  3). 
In  other  words,  two  mutually  exclusive  attrition 
outcomes  are  possible  at  the  first  assigned  facility. 
The  trainee  might  be  (a)  separated  from  the  ATCS 
occupation,  or  (b)  offered  the  option  to  switch  to  a 
less  demanding  facility,  such  as  a  lower-level  terminal 
facility  or  a  Flight  Service  Station  (FSS).  Attrition,  in 
either  form,  represents  a  significant  economic  cost  to 
the  agency.  Attrition  from  ATCS  training  has  been 
and  continues  to  be  a  significant  concern  to  the  U.S. 
Congress  as  well  (U.S.  Congress,  1976;  General  Ac¬ 
counting  Office,  1989b).  Attrition  rates  for  the  two 
evaluation  groups  were  compared,  using  Fisher’s  Z- 
test  under  the  following  hypothesis: 

M nATCTC  and  FAA  Academy  graduate  attri¬ 
tion  rates,  defined  as  separations  from  the  occupation 
and  switches from  the first  assigned  en  route facility,  will 
be  equal. 

Performance  ratings 

First,  the  overall  internal  consistency  of  the  ratings 
provided  by  instructors,  incumbents,  and  supervisors 
was  estimated  for  each  major  domain  of  the  rating 
instrument  as  a  measure  of  instrument  internal  reli¬ 
ability.  Second,  scale  scores  were  computed  by  aver- 
aging  valid  responses  across  items  comprising  the 
scale.  Third,  scale  scores  were  correlated  by  rating 
source  to  assess  inter-rater  reliability.  Finally,  do¬ 
main  scores  for  each  group  were  compared  by  one¬ 
way  analysis  of  variance  to  test  the  following 
hypothesis: 

The  mean  ratings  of  technical  skill,  technical 
knowledge,  teamwork,  and  overall  potential  to  succeed 
in  the  occupation  given  by  supervisors  and  OJT instruc- 
to  rs  of  MnA  TCTC  and  FAA  Academy  graduates,  and  by 
the  graduates  themselves,  will  be  equal. 

As  discussed  previously,  institutions  participating 
in  the  CTI/ATCS  program  had  raised  concerns  about 
how  two  factors  might  influence  the  subjective 
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performance  ratings.  The  first  factor  was  acceptance 
in  the  facility,  assessed  by  the  incumbent’s  and  other’s 
perceptions.  The  second  factor  was  the  delay  between 
graduation  and  hiring  for  MnATCTC  graduates. 
Therefore,  three  additional  analyses  of  variance  were 
conducted,  controlling  separately  and  jointly  for  (a) 
the  degree  to  which  the  incumbent  felt  accepted  in 
the  facility,  (b)  the  degree  to  which  the  supervisor 
perceived  the  controller  as  being  accepted  in  the 
facility,  and  (c)  hiring  delay. 

Costs  and  benefits 

As  no  formal  statistical  analyses  are  associated  with 
cost-benefit  comparisons,  the  research  issues  are 
framed  as  questions,  rather  than  formal  hypotheses. 
The  first  cost-benefit  analysis  focused  on  a  compari¬ 
son  of  the  cost-per-hire  between  the  MnATCTC  and 
the  FAA  Academy  programs.  The  research  question 
addressed  in  this  first  analysis  was: 

Qj:  At  what  point  might  the  MnATCTC  cost-per- 
hire  be  equal  to  or  less  than  the  cost-per-controller  at  the 
FAA  Academy? 

The  second  analysis  evaluated  the  ratio  of  accrued 
and  projected  costs,  and  benefits  for  the  MnATCTC 
program  to  address  the  research  question: 

Q^:  At  what  point  are  the  benefits  accruing  from  the 

MnATCTC  program  likely  to  balance  or  exceed  the 
direct  and  indirect  costs  of  the  program,  e.g,  return  one 
dollar  or  more  in  savings  to  the  FAA  for  each  dollar 
invested? 

RESULTS 

Employee  diversity 

There  were  no  significant  differences  by  partici¬ 
pating  institution  in  the  representation  of  minorities 
(Table  4).  However,  there  were  significantly  more 
women  in  the  MnATCTC  sample  (39.7%)  than  in 
the  FAA  Academy  sample  (17.2%;  Z  =  3.67,  p  < 
.001).  As  a  result,  the  proportion  of  male  non-mi¬ 
norities  in  the  MnATCTC  sample  was  significantly 
smaller  (53-7%)  than  in  the  Academy  graduates 
(67.2%;  Z  =  -2.55,  p  <  .01),  apparently  due  to  the 
larger  proportion  of  non-minority  females  in  the 
MnATCTC  sample. 

Field  training  progress 

Descriptive  statistics  for  the  days,  hours  on-the- 
job  training,  and  indicator  of  performance  (IP)  rating 
for  each  phase  of  en  route  training  by  program  are 


presented  in  Table  10.  Fewer  days  in  phase  and  fewer 
hours  of  OJT  represent  better  performance;  con¬ 
versely,  a  higher  IP  indicates  better  performance.  On 
one  hand,  the  one-way  analyses  of  variance  of  raw 
training  data  by  group  indicated  that  MnATCTC 
graduates  did  significantly  better  than  the  compari¬ 
son  group  of  FAA  Academy  graduates  on  the  follow¬ 
ing  training  measures;  number  of  days  and  hours  of 
OJT  in  Phase  VI;  hours  of  OJT  in  Phase  VIII;  days 
in  Phase  IX,  with  and  without  adjustment  for  the 
number  of  sectors  on  which  training  was  provided; 
days  and  hours  ofOJT  in  PhaseXI;  and  days  in  Phase 
XIII.  On  the  other  hand.  Academy  graduates  ap¬ 
peared  to  do  better  than  MnATCTC  graduates  on 
the  following  training  measures:  Phase  VII  days  and 
hours  of  OJT;  hours  of  OJT  in  Phase  IX,  with  and 
without  adjusting  for  the  number  of  sectors  on  which 
training  was  provided;  and  hours  of  OJT,  adjusted 
for  number  of  training  sectors,  in  PHASEXand  Phase  XI. 

Analyses  based  on  raw  data,  without  consideration 
of  inter-facility  differences,  are  misleading,  however. 
As  noted  above,  previous  research  has  found  substan¬ 
tive  differences  on  training  progress  measures  be¬ 
tween  en  route  facilities  not  attributable  to  individual 
differences  in  the  abilities  of  trainee  controllers  (Of¬ 
fice  of  the  Deputy  Associate  Administrator  for  Ap¬ 
praisal ,  1 989) .  A  more  realistic  appraisal  of  the  progress 
of  MnATCTC  and  FAA  Academy  graduates  is  pro¬ 
vided  by  an  analysis  of  training  measures  standard¬ 
ized  with  respect  to  facility  historical  means  and 
standard  deviations  for  each  phase.  For  example,  the 
days  in  Phase  IX  for  a  MnATCTC  or  FAA  Academy 
graduate  assigned  to  Minneapolis  center  would  be 
standardized  with  respect  to  the  historical  mean  and 
standard  deviation  of  days  in  Phase  IXat  that  facility. 

The  results  of  the  comparison  of  MnATCTC  to 
FAA  Academy  graduates,  using  standardized  training 
measures,  are  presented  in  Table  11.  A  negative 
standardized  score  for  the  days  and  hours  OJT  in 
phase  indicate  better  performance  than  average.  That 
is,  a  negative  score  indicates  that  a  person  took  fewer 
days  or  hours  of  OJT  than  average.  A  positive  stan¬ 
dardized  score  for  IP  indicates  a  rating  higher  than 
average. 

On  one  hand,  graduates  from  the  MnATCTC 
program  performed  statistically  better  than  FAA  Acad¬ 
emy  graduates  on  six  standardized  training  measures. 
The  graduates  from  the  MnATCTC  program  re¬ 
quired  statistically  fewer  standardized  days  (0.02 
standard  deviations  above  facility  average)  in  the 
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Table  10.  En  route  days  in  phase,  hours  OJT  in  phase,  and  phase  Indicator  of  Performance  for  MnATCTC  and  Academy 
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Preliminary  Nonradar/Radar  Associate  phase  of  field 
training  (Phase  VII)  than  FAA  Academy  graduates 
(0. 14  standard  deviations  more  than  facility  averages; 
F{1,  279)  -  5.68,  p  <  .05).  MnATCTC  graduates 
required  statistically  fewer  standardized  days  of  train¬ 
ing  in  Phase  IX  (-0.85  standard  deviation  units  fewer 
than  average)  than  FAA  Academy  graduates  (-0.13 
standard  deviation  units  fewer  than  average;  /*(! ,  1 1 0) 
=  10.34,  p  <  .01).  The  same  pattern  held  when  the 
days  in  Phase  IX  were  adjusted  for  the  number  of 
sectors  on  which  controllers  trained  (MnATCTC  =  - 
0.84  standard  deviations  fewer  than  average,  versus  - 
0.13  standard  deviations  fewer  than  average  for 
Academy  graduates;  /'(1, 110)  =  10.15,  p  <  .01). 
MnATCTC  graduates  also  took  fewer  standardized 
days  (0,02  standard  deviations  more  than  facility 
averages)  in  Final  Radar  Association  Qualification 
training  (Phase  X)  than  FAA  Academy  graduates 
(0.50  standard  deviations  more  than  facility  averages; 
F(l,  200)  =  8.60,  p  <  .01).  Adjusting  the  days  in 
Phase  X  training  for  the  number  of  sectors  on  which 
controllers  were  trained  did  not  change  this  pattern 
of  results;  MnATCTC  required  fewer  adjusted  days 
in  training  (just  0. 1 1  standard  deviations  more  than 
facility  averages)  than  FAA  Academy  graduates  (0.58 
standard  deviations  more  than  facility  averages; 
F'(l,184)  =  5.84,/^  <  .05).  The  MnATCTC  graduates 
also  required  statistically  fewer  standardized  days  in 
Radar  Controller  Training  (Phase  XI)  than  FAA 
Academy  graduates.  MnATCTC  graduates  required 
0.51  standard  deviations  more  than  facility  averages 
to  complete  Phase  XI,  while  FAA  Academy  graduates 
required  0.71  standard  deviations  more  than  facility 
averages  in  the  same  training  (F'(  1 ,2 1 3)  =  5. 1  Syp  <  .05). 

On  the  other  hand,  FAA  Academy  graduates  per¬ 
formed  better  than  MnATCTC  graduates  on  three 
standardized  training  measures.  FAA  Academy  gradu¬ 
ates  earned  higher  standardized  IPs,  relative  to  facil¬ 
ity  averages,  in  Phase  VI  (0.33  standard  deviations 
above  facility  averages)  than  did  MnATCTC  gradu¬ 
ates  (just  0.03  standard  deviations  above  facility 
averages;  F'(l,237)  =  5.05,  / <  .05).  The  FAA  Acad- 
emy  graduates  took  fewer  standardized  days  (0.24 
standard  deviations  more  than  facility  averages)  in 
the  Initial  Radar  Association  Qualification  training 
phase  (Phase  VIII)  than  MnATCTC  graduates  (0.50 
standard  deviations  more  than  facility  averages;  /*(!, 
282)  =  5.22,  p  <  .05).  The  FAA  Academy  graduates 
required  statistically  fewer  standardized  hours  of  OJT 
in  Phase  X  (0.44  standard  deviations  more  than 


facility  averages),  adjusted  for  the  number  of  sectors 
on  which  they  trained,  than  did  MnATCTC  gradu¬ 
ates  (0.71  standard  deviations  above  facility  averages; 

F{ly  184)  =  4.29,/><.05). 

Training  outcomes 

Attrition 

Outcomes  at  the  first  assigned  field  facility  by 
school  are  presented  in  Table  12.  Available  informa¬ 
tion  indicated  that  10  (7.4%)  MnATCTC  graduates 
had  attrited  from  the  FG-21 52  occupation  as  of  June 
1995.  Just  6  (3.8%)  FAA  Academy  graduates  had 
attrited  from  their  first  assigned  facility  as  of  June 
1995.  The  attrition  rates  were  not  significantly  dif¬ 
ferent  (Z=  0.91,  ns).  Significantly  more  FAA  Acad¬ 
emy  graduates  switched  options  or  facilities  {N  =  7; 
4.5%),  prior  to  reaching  the  full  performance  level 
(FPL)  at  their  first  assigned  facility,  than  did 
MnATCTC  graduates  (A^=  l,or0.7%;Z=  1.99, 
p  <  ‘05).  Statistically  more  MnATCTC  graduates 
(75.0%)  were  still  in  training  at  their  first  assigned 
facility  than  FAA  Academy  graduates  (22.9%;  Z  = 
8.91,  p  <  .001).  Overall,  significantly  fewer 
MnATCTC  had  achieved  FPL  as  of  June  1995  (A^= 
23,  or  16.9%)  than  FAA  Academy  graduates  {N  = 
108,  or  68.8%;  Z=  -8.91,/ <  .001). 

Survival  analysis 

Survival  analysis  was  also  used  to  examine  the 
proportions  ofFAA  Academy  and  MnATCTC  gradu¬ 
ates  remaining  in  the  first  assigned  facility,  that  is, 
who  had  neither  been  separated  from  the  occupation 
nor  had  switched  facilities  or  options  as  ofjune  1995. 
Survival  analysis  is  a  useful  technique  for  examining 
the  interval  between  two  events,  such  as  enrollment 
in  field  training  and  attrition,  when  the  second  event 
(attrition)  does  not  necessarily  happen  to  everyone, 
and  when  subjects  are  observed  for  different  periods 
of  time  (Norusis,  1990),  as  is  the  case  in  this  evalua¬ 
tion.  In  survival  analysis,  the  overall  period  of  obser¬ 
vation  is  subdivided  into  intervals.  For  this  summative 
evaluation  of  the  MnATCTC  program,  the  period 
between  enrollment  on  the  job  for  each  subject  and 
June  30,  1995,  or  attrition  from  the  first  facility,  was 
the  observation  period.  The  observation  period  was 
subdivided  into  one  month  intervals  using  the  SPSS 
SURVIVAL  procedure.  For  each  one  month  interval, 
all  subjects  who  were  observed  for  at  least  that  long 
were  used  to  calculate  the  probability  of  attrition 
occurring  in  that  interval  by  the  SURVIVAL 
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(Table  1 1  continued) 
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Table  12.  Outcomes  at  first  assigned  facility 


Outcome 

MnATCTC 

FAA 

Academy 

Total 

Attrited  from  2152 

10 

6 

16 

(7.4%) 

(3.8%) 

(5.5%) 

Moved  from  Istfac 

1 

7 

8 

(0.7%) 

(4.5%) 

(2.7%) 

Still  developmental 

102 

36 

138 

(75.0%) 

(22.9%) 

(47.1%) 

Made  FPL 

23 

108 

131 

(16.9%) 

(68.8%) 

(44.7%) 

Total 

136 

157 

293 

Program 
°  MnATCTC 
o  FAA  Academy 


Months  from  OJT  start  to  attrition 
Figure  1.  MnATCTC  and  FAA  Academy  attrition/survival  analysis 


procedure.  The  result  is  an  estimate,  for  each  group, 
of  the  cumulative  proportion  of  graduates  remaining 
at  the  first  assigned  facility  at  one,  two,  three  and  so 
on  months  after  enrollment  in  field  training.  The 
SPSS  SURVIVAL  procedure  uses  the  Wilcoxon  or 
Gehan  statistic  to  test  the  hypothesis  that  the  survival 
distributions  are  the  same  for  MnATCTC  and  FAA 
Academy  samples  (Norusis,  1990,  p.  244). 


The  results  of  the  survival  analysis  are  presented  in 
Figure  1.  More  than  90%  of  both  FAA  Academy  and 
MnATCTC  graduates  had  survived  at  the  first  as¬ 
signed  facility,  as  would  be  expected  from  the  simple 
rate  of  attrition  analysis  reported  above.  The  survival 
distributions  were  not  statistically  different  (Wilcoxon 
=  2.987,  df=  l,p  =  .084,  ns)  for  the  two  groups.  This 
analysis  suggested  that,  after  taking  into  account  the 
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different  lengths  of  time  the  groups  had  been  at  their 
first  assigned  facilities,  there  were  no  statistically 
significant  differences  in  the  proportions  of 
MnAT CT C  and  FAA  Academy  graduates  that  would 
be  expected  to  remain  (e.g. ,  neither  be  separated  from 
the  occupation  nor  switch  option  or  facility)  at  the 
first  assigned  facility  over  time. 

FPL  Certification 

Time  to  FPL 

Just  23  (16.9%)  MnATCTC  graduates  were  certi¬ 
fied  as  FPLs  at  their  first  assigned  field  facility  as  of 
June  1995.  In  contrast,  68.8%  of  the  FAA  Academy 
graduates  had  attained  FPL  certification  by  June 
1995.  This  is  consistent  with  the  greater  time  at  the 
first  assigned  facility  for  FAA  Academy  graduates: 
with  longer  times  in  the  facility,  a  larger  proportion 
of  FAA  Academy  graduates  would  be  expected  to 
have  completed  the  field  training  sequence.  The 
average  number  of  years  to  certification,  as  shown  in 
Table  13,  was  not  significantly  different  between  the 
two  programs.  MnATCTC  graduates  required  about 
2.82  (SD  =  0.59)  years  to  certify,  in  comparison  to 
3.18  (SD  =  0.53)  years  for  FAA  Academy  graduates 
(F'(l,130)  =  0.00,  ns).  Times  to  FPL  were  also  stan¬ 
dardized,  with  respect  to  historical  facility  averages, 
and  compared.  Both  groups  required  slightly  more 
time  to  FPL  than  historical  averages  (0.34  standard 
deviations  more  than  average  for  MnATCTC  gradu¬ 
ates,  compared  with  0.55  standard  deviations  more 
than  average  for  FAA  Academy  graduates).  However, 
the  standardized  times  to  FPL  were  not  statistically 
different  for  the  two  groups  (^(1,  130)  =  0.22,  ns). 


Accession  to  FPL 

Survival  analysis  was  used  to  evaluate,  from  a 
different  perspective,  the  proportion  of  surviving 
graduates  by  program  and  option  still  in  training 
(developmental  status),  as  of  June  1995,  relative  to 
their  enrollment  date.  The  terminating  event  in  this 
analysis  was  making  FPL.  The  analysis  provides  com¬ 
parative  data  about  when  program  graduates  made 
FPL,  taking  into  account  the  different  lengths  of 
observation.  The  results  of  this  second  survival  analy¬ 
sis  for  the  Academy  and  MnATCTC  graduates  who 
did  not  attrite  in  the  enroute  option  are  presented  in 
Figure  2.  Taking  into  account  the  differing  amounts 
of  time  in  the  field,  the  survival  distributions  for  the 
two  groups  are  not  statistically  different  (Wilcoxon  = 
0.039,  df=  1,  ns).  This  analysis  suggests  that,  with 
time,  MnATCTC  graduates  can  be  expected  to  achieve 
FPL  certification  at  about  the  same  time  and  at  about 
the  same  rates  as  FAA  Academy  graduates. 

Performance  ratings 

Reliability 

Scale  scores  for  4  domains  were  computed  from 
the  returned  surveys:  TEAMWORK  (15  items);  degree  of 
acceptance  (acceptance)  in  the  facility  (5  items); 
TECHNICAL  SKILL  (11  items);  and  TECHNICAL  KNOWL¬ 
EDGE  (11  items).  Scale  scores  were  computed  as  the 
average  of  valid  responses  to  the  items  comprising  a 
scale.  A  scale  score  of  1  indicated  a  low  degree,  2  an 
acceptable  degree,  3  a  higher  degree,  and  4  the  highest 
degree  of  acceptance,  teamwork,  technical  skill, 
or  technical  knowledge.  Estimates  of  Cronbach’s 
alpha  (a),  a  measure  of  internal  consistency,  for  the 


Table  13.  Years  and  standardized  years  to  full  performance  level  (FPL)  in  the  en  route 
option 


MnATCTC 

FAA  Academy 

M 

SD  N 

M 

SD  N 

F 

Years  to  FPL 
ZYears  to  FPL® 

2.82 

0.34 

0.59  23 

0.87  23 

3.18 

0.55 

0.53  108 

0.70  108 

0.00 

0.22 

Notes.  Years  fo  FPL  standardized  with  respect  to  historical  means  and  standard  deviations  for  faciiitv 
type  and  level  to  which  person  was  assigned 
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Program 

°  MnATCTC 
“  FAA  Academy 


Moiitlis  from  startiiig  O TT  to  FPL 

Figure  2.  MnATCTC  and  FAA  Academy  accession  to  FPL  over 
time  analysis 


TEAMWORK  scale  were  .94  for  OJT  instructors,  .94 
for  controllers,  and  .96  for  supervisors.  Internal 
consistency  estimates  for  the  5-item  acceptance 
scale  were  lower  and  barely  acceptable,  ranging 
from  .42  for  supervisors  to  .59  for  controllers.  The 

TECHNICAL  SKILL  and  TECHNICAL  KNOWLEDGE  Scale 
internal  consistency  estimates  were  above  .90  for 
all  three  rating  sources. 

The  degree  of  inter-rater  reliability  on  the  scales 
was  assessed  by  computing  correlations  between  in¬ 
structors,  controllers,  and  supervisors.  Inter-rater 
correlations  of  instructors  with  controller  and  super¬ 
visor  ratings  were  generally  low  for  teamwork,  tech¬ 
nical  SKILL,  TECHNICAL  KNOWLEDGE,  and  ratings  of 
potential  to  succeed  (potential),  as  shown  in  Table 
14.  Correlations  of  controller  with  supervisor  ratings 
were  moderate  for  TEAMWORK,  technical  skill,  and 
TECHNICAL  KNOWLEDGE.  The  correlation  between  con¬ 
troller  self-  and  supervisory  evaluation  of  POTENTIAL 
to  succeed  in  the  occupation  was  good  (r  =  .72,  p  < 


,001).  Overall,  the  pattern  of  results  indicated  a 
relatively  low  degree  of  agreement  between  raters  by 
source.  Therefore,  the  ratings  data  were  analyzed  by 
rating  source. 

OJT  Instructor 

The  analysis  of  OJT  instructor  ratings  by  perfor¬ 
mance  domain  by  school  is  presented  in  Table  15. 
Instructor  ratings  were  returned  for  53  MnATCTC 
and  61  FAA  Academy  graduates.  The  instructor 
evaluation  of  the  degree  to  which  the  person  rated  was 
accepted  (acceptance)  in  the  facility  did  not  differ  by 
school  (F(l  ,1 13)  =  1 .06,  ns).  That  is,  OJT  instructors 
perceived  MnATCTC  graduates  as  being  as  well 
accepted  in  the  facility  as  FAA  Academy  graduates. 
The  mean  instructor  ratings  on  teamwork  for 
MnATCTC  and  FAA  Academy  graduates  were  not 
statistically  different  (/"(I,!  13)  =  0.65,  ns).  In  other 
words,  OJT  instructors  perceived  the  teamwork  of 
MnATCTC  graduates  as  being  about  the  same  as  that 
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Table  14  Performanoe  ratings  dimension  inter-rater  oorrelations  by  rating 
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of  FAA  Academy  graduates.  However,  OJT  instruc¬ 
tors  rated  the  technical  skill  of  MnATCTC  gradu¬ 
ates  as  being  lower  than  that  of  FAA  Academy 
graduates  <F(1.113)  =  7.17,  p  <  .01).  These  same 
instructors  also  evaluated  the  technical  knowledge 
of  MnATCTC  graduates  as  being  lower  than  that  of 
FAA  Academy  graduates  (i^(l,l  13)  =  6.33,/  <  .05). 
However,  the  overall  potential  of  MnATCT C  gradu¬ 
ates  to  succeed  in  the  occupation  (M  =  79.65,  SD  = 
18.74 on  a 40  to  100  scale)  was  not  rated  significantly 
lower  than  that  of  FAA  Academy  graduates  (Af  = 
85.77,  SD  =  15-95;  =  3.48,  ns). 

Controller 

Overall,  die  mean  scale  scores  for  FAA  Academy 
graduates  across  all  five  rating  dimensions  were  sta¬ 
tistically  higher  than  the  mean  scale  scores  for 
MnATCTC  controllers  when  computed  on  self-rat¬ 
ings  (Table  1 6) .  The  average  sdf-rating  on  teamwork 
for  FAA  Academy  graduates  was  2.68  (on  a  1  to  5 


scale),  compared  with  a  mean  of  2.42  for  MnATCTC 
self-evaluations  of  teamwork  (F( 1,228)  =  9.71,/  < 
.01).  FAA  Academy  graduates  felt  themselves  to  be 
better  accepted  at  the  facility  (acceptance:  M-  2.85, 
SD  =  0.56)  than  did  MnATCTC  graduates  (Af  = 
2.60,  SD  =  0.69;  i^(1.228)  =  9.18,  /  <  .01).  FAA 
Academy  graduates  also  rated  their  technical  skill 
more  highly  {M  =  2.33,  SD  =  0.55)  than  did 
MnATCTCgraduates  (A/=  1 .98,  SD  =  0.75;  F(1 ,225) 
=  16.72,  /  <  .001).  Similarly,  FAA  Academy  gradu¬ 
ates  rated  their  technical  knowledge  of  air  traffic 
control  more  highly  (Af  =  2.69,  SD  =  0.50  on  a  1  to 
5  scale)  dian  did  MnATCTC  graduates  {M  =  2.36, 
SD  =  0.70;  F(l,227)  =  l6.48,/<. 001).  Finally,  FAA 
Academy  graduates  rated  themselves  as  having  greater 
TOTENTIAL  to  succeed  in  the  occupation  {M  =  91.45, 
SD  =  11.73  on  a  40-100  scale)  than  did  the 
MnATCTC  graduates  (A/  =  84.22,  SD  =  18.77; 
F(l,229)  =  12.44,/ <  .001). 


Fable  15.  OJT  Instructor  (OJT-I)  ratings  on  performance  dimensions 


MnATCTC 

FAA  Academy 

M 

SD 

N 

M 

SD 

N 

F 

Teamwork 

Acceptance 

Technical  skill 
Technical  knowledge 
Potential 

2.29 

2.70 

1.83 

2.28 

79.65 

0.76 

0.65 

0.65 

0.66 

18.74 

53 

53 

53 

53 

52 

2.40 

2.81 

2.16 

2.57 

85.77 

0.72 

0.50 

0.66 

0.58 

15.95 

61 

61 

61 

61 

60 

0.65 

1.06 

7.17** 

6.33* 

3.48 

*p<.05,  ** 

p<  .01, 

***p<.001 

Table  16.  Controller  self-ratings  on  performance  dimensions 

MnATCTC 

FAA  Academy 

M 

SD 

N 

M 

SD 

N 

F 

Teamwork 

Acceptance 

Technical  skill 
Technical  knowledge 
Potential 

2.42 

2.60 

1.98 

2.36 

84.22 

0.75 

0.69 

0.75 

0.70 

18.77 

109 

109 

106 

108 

111 

2.68 

2.85 

2.33 

2.69 

91.45 

0.51 

0.56 

0.55 

0.50 

11.73 

120 

120 

120 

120 

119 

9.71** 

9.18** 

16.72*** 

16.48*** 

12.44*** 

*p<.05, 

'p<.01. 

***p<.001 
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Supervisor 

The  supervisor  ratings  (Table  17)  followed  a  simi¬ 
lar  pattern  as  the  controller  self-ratings:  the  mean 
ratings  assigned  by  supervisors  for  FAA  Academy 
graduates  were  statistically  greater  than  the  mean 
ratings  for  MnATCTC  graduates  across  all  five  rating 
dimensions.  The  mean  scale  score  of  supervisory 
ratings  on  teamwork  fo  r  FAA  Academy  graduates  (M 
=  2.48,  SD  =  0.62  on  a  1  to  5  scale)  were  statistically 
greater  than  those  for  MnATCTC  graduates  {M  = 
2. 1 1, 5Z)  =  0.69;  F{IA58)  =  12.93,/ <  .001).  Super¬ 
visors  perceived  FAA  Academy  graduates  as  being 
better  accepted  at  the  facility  (acceptance:  Af  =  2.88, 
SD  =  0.62)  than  MnATCTC  graduates  {M=  2.65,  SD 
=  0.61,  F'(l,157)  =  5.22,  p  <  .05).  The  technical 
SKILL  of  FAA  Academy  graduates  {M  =  2.09,  SD  = 
0.58)  was  rated  more  highly  by  supervisors  than  was 
that  of  MnATCTC  graduates  {M  =  1.86,  SD  =  0.63; 
7^(1, 155)  =  5.19,  /  <  .05).  The  mean  technical 
knowledge  ratings  by  supervisors  followed  the  same 
pattern,  with  FAA  Academy  graduates  being  rated 
more  highly,  on  average  (Af  =  2.56,  SD  =  0.55)  than 
MnATCTC  graduates  (Af  =  2.20, 5D  =  0.67;  /*(  1 , 1 57) 
=  13.16,  p  <  .001).  Finally,  supervisors  saw  more 
potential  to  succeed  in  the  ATCS  occupation,  on 
average,  in  FAA  Academy  graduates  (Af  =  86.08,  SD 
=  15.65,  on  a  40-100  scale)  than  in  MnATCTC  gradu¬ 
ates  (Af=  80.17, 18.47; /^(l, 156)  =  4.65,/ <. 05). 

Influence  of  degree  of  acceptance  and  delay  on 
supervisor  ratings 

As  noted  previously,  institutions  participating  in 
the  CTI/ATCS  program  had  raised  concerns  about 
the  potential  influence  of  the  degree  to  which  a 
program  graduate  was  accepted  at  the  local  facility 
on  supervisory  ratings  of  skill,  knowledge,  and 


performance.  Concerns  about  the  impact  of  substan¬ 
tial  hiring  delays  on  those  ratings,  due  to  skill  and 
knowledge  decay  over  time,  were  also  expressed  by 
institutional  representatives.  Therefore,  supplemen¬ 
tal  analyses  of  covariance  were  conducted,  in  which 
supervisor  ratings  were  analyzed  after  taking  into 
account  the  effects  of  acceptance  and  hiring  delay  as 
covariates.  For  these  analyses,  the  degree  of  accep¬ 
tance  in  the  facility  was  taken  from  the  controller’s 
perspective.  Procedurally,  the  controller’s  perceived 
degree  of  acceptance  in  the  facility  and  the  time 
between  graduation  and  hiring  were  processed  as 
covariates  before  the  main  effect  of  program 
(MnATCTC  versus  FAA  Academy)  was  analyzed 
using  the  SPSS  ANOVA  command.  Instructor  evalu¬ 
ations  were  not  addressed  in  this  analysis,  as  the 
concern  expressed  by  institutional  representatives 
focused  on  supervisors. 

Degree  of  acceptance.  The  results  of  the  analysis 
of  supervisor  ratings  across  performance  dimensions, 
controlling  for  the  incumbent  specialist’s  perceived 
degree  of  acceptance  in  the  facility,  are  presented  in 
Table  18.  The  first  dimension  considered  was  team¬ 
work.  The  total  variability  in  the  teamwork  rating  is 
subdivided  in  the  analysis  into  four  components  in 
three  steps.  First,  the  amount  of  the  variability  attrib¬ 
utable  to  controllers’  ratings  of  ACCEPTANCE  in  the 
facility  was  computed: 

Total  variability  =  variability  due  to 

ACCEPTANCE  +  remaining  variability. 

A  test  was  computed  to  determine  if  the  amount  of 
variability  explained  by  degree  of  acceptance  (the 
covariate)  was  statistically  significant.  Second,  the 


Table  17.  Supervisor  ratings  on  performance  dimensions 


Teamwork 
Acceptance 
Technical  skill 
Technical  knowledge 
Potential 


MnATCTC  FAA  Academy 


M 

SD 

N 

M 

SD 

N 

F 

2.11 

0.69 

64 

2.48 

0.62 

95 

12.93*** 

2.65 

0.61 

63 

2.88 

0.62 

95 

5.22* 

1.86 

0.63 

61 

2.09 

0.58 

95 

5.19* 

2.20 

0.67 

63 

2.56 

0.55 

95 

13.16*** 

80.17 

18.47 

64 

86.08 

15.65 

93 

4.65* 

*p  <  .05,  **p<  .01,  ***p<  .001 
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Table  18.  Supervisor  ratings  on  performance  dimensions,  controlling  for  controller’s  perceived  degree  of 
ACCEPTANCE  in  the  facility 

Program 
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amount  of  remaining  variability  explained  by  or 
attributable  to  the  differences  between  programs,  as 
the  main  effect,  was  computed: 

Total  variability  =  variability  due  to  ACCEPTANCE  + 
variability  due  to  PROGRAM  +  leftover  variability. 

A  test  is  computed  to  determine  if  the  amount  of 
variability  explained  by  program,  as  the  main  effect, 
was  statistically  significant.  Finally,  the  amount  of 
variability  in  teamwork  ratings  attributable  to  the 
joint  effects  of  ACCEPTANCE  and  PROGRAM,  as  the 
overall  model  for  the  relationships  between  TEAMWORK 
ratings,  ACCEPTANCE,  and  program,  was  computed: 

Total  variability  =  variability  due  to  acceptance  + 
variability  due  to  PROGRAM  +  variability 
due  to  both  +  residual  variability. 

A  final  test  was  computed  to  determine  if  the 
amount  of  variability  explained  jointly  by  acceptance 
and  program  was  statistically  significant. 

The  mean  supervisor’s  rating  of  teamwork  for 
MnATCTC  graduates  was  2.08,  compared  with  2.48 
for  FAA  Academy  graduates.  The  covariate  in  the 
analysis  was  each  controller’s  scale  score  for  the 
degree  to  which  he  or  she  felt  accepted  at  the  facility 
(acceptance).  The  covariate  accounted  for  a  statisti¬ 
cally  significant  portion  of  the  overall  variability  in 
the  supervisory  ratings  of  TEAMWORK  (A  =  43. 1 1 ,  />  < 
.001).  Program  also  accounted  for  a  statistically 
significant  portion  of  the  overall  variability  in  TEAM¬ 
WORK  supervisory  ratings  {F  =  8.86,  p  <  .01).  The 
degree  of  acceptance  and  program  also  jointly  ac¬ 
counted  for  a  statistically  significant  portion  of  vari¬ 
ance  in  supervisory  ratings  of  teamwork  (F=  25.99, 
p  <  .001).  This  pattern  of  results  suggests  that,  even 
after  accounting  for  the  effects  of  how  well  the 
controller  felt  accepted  at  the  facility,  the  MnATCTC 
graduates  were  still  rated  lower  on  teamwork  than 
FAA  Academy  graduates. 

Analysis  of  the  supervisor’s  TECHNICAL  SKILL  rat¬ 
ings  presents  a  different  picture.  The  mean  rating  for 
MnATCTC  graduates  was  1 .84,  compared  with  2.09 
for  FAA  Academy  graduates  on  the  technical  skill 
dimension.  The  degree  to  which  the  incumbent  con¬ 
troller  felt  accepted  at  the  facility  (acceptance)  ac¬ 
counted  for  a  statistically  significant  portion  of  the 
overall  variability  in  supervisory  ratings  of  technical 
SKILL  {F  =  7.85,  p  i  .01).  However,  program 


(MnATCTC  or  FAA  Academy)  did  not  significantly 
affect  supervisory  ratings  of  TECHNICAL  skill.  This 
analysis  suggests  that  differences  in  the  mean  ratings 
of  the  technical  skill  of  MnATCTC  and  FAA  Acad¬ 
emy  graduates  may  be  attributable  to  differences  in 
the  degree  to  which  controllers  were  accepted  at  the 
facility  rather  than  to  where  they  were  initially  trained. 

Analysis  of  the  supervisory  ratings  of  technical 
KNOWLEDGE  present  a  pattern  similar  to  that  of  the 
teamwork  ratings.  The  mean  rating  of  MnATCTC 
graduate  technical  knowledge  by  supervisors  was 
2. 19,  compared  with  2.56  for  FAA  Academy  gradu¬ 
ates  on  a  1  (low)  to  5  (high)  point  scale.  The  degree 
of  acceptance,  from  the  controller’s  perspective, 
accounted  for  a  statistically  significant  portion  on  the 
variance  in  supervisory  ratings  of  TECHNICAL  KNOWL¬ 
EDGE  (A  =  23.28,  p  <  .001).  However,  even  after 
accounting  for  the  effects  of  acceptance,  the  pro¬ 
gram  at  which  the  incumbent  was  initially  trained 
(program)  still  accounted  for  a  statistically  signifi¬ 
cant  portion  of  the  variance  in  supervisory  ratings  of 
technical  knowledge  {F  =  8.84,  p  <  .01).  This 
pattern  suggests  that  differences  in  ratings  of  techni¬ 
cal  KNOWLEDGE  for  MnATCTC  and  FAA  Academy 
graduates  cannot  be  explained  away  merely  as  a 
consequence  of  MnATCTC  graduates  being  less  well 
accepted  at  the  facility.  Rather,  the  difference  in  super¬ 
visory  ratings  may  reflect  real  differences  in  the  air  traffic 
control  technical  knowledge  of  the  two  groups. 

The  pattern  of  results  for  the  analysis  of  supervi¬ 
sory  ratings  of  incumbent  potential  to  succeed  is 
similar  to  that  of  ratings  of  TECHNICAL  SKILL.  The 
mean  rating  of  MnATCTC  graduate  potential  to 
succeed  was  79.94,  compared  with  86.08  for  FAA 
Academy  graduates.  The  degree  to  which  controllers 
felt  accepted  at  the  facility  (acceptance)  accounted 
for  a  statistically  significant  portion  of  the  overall 
variability  in  ratings  of  potential  to  succeed  (F  = 
l4.31,/><  .001).  However,  program  (MnATCTC  or 
FAA  Academy)  did  not  account  for  any  significant 
variance  {F  =  2.48,  ns).  This  analysis  suggests  that 
differences  in  the  supervisory  ratings  of  POTENTIAL  to 
succeed  might  have  been  influenced  by  the  degree  to 
which  the  new  controllers  were  accepted  at  the  facil¬ 
ity,  but  not  by  the  program  from  which  the  controller 
graduated.  In  other  words,  persons  perceiving  them¬ 
selves  as  less  well  accepted  at  the  facility  were  also  seen 
as  having  less  potential  to  succeed  in  the  ATCS 
occupation  by  their  supervisors,  regardless  of  the 
person’s  hiring  source  (FAA  Academy  or  MnATCTC). 
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Hiring  delay.  The  second  covariate  analyzed  was 
the  delay  between  graduation  and  hiring.  FAA  Acad¬ 
emy  graduates  experienced  very  little  delay  between 
Academy  graduation  and  starting  OJT  at  their  first 
facility  (Af  =  0.23  months).  MnATCTC  graduates,  in 
contrast,  experienced  average  delays  of  6  months 
between  graduation  in  Minnesota  and  starting  OJT 
(Af  =  6.02  months,  SD  =  3.07).  It  was  hypothesized 
by  CTI/ATCS  representatives  that  the  hiring  delay 
would  lead  to  a  time-based  degradation  of  skills  and 
knowledge.  As  a  consequence,  MnATCTC  graduates 
might  receive  lower  ratings  than  FAA  Academy  gradu¬ 
ates  who  had  not  experienced  those  delays  and  atten¬ 
dant  knowledge  and  skill  losses.  As  with  the  degree  of 
ACCEPTANCE,  analysis  of  covariance  was  used  to  ex¬ 
plore  the  degree  to  which  hiring  delays  influenced 
supervisory  ratings  of  teamwork,  technical  skill, 

TECHNICAL  KNOWLEDGE,  and  POTENTIAL  tO  SUCCeed  in 
the  ATCS  occupation. 

The  results  of  this  analysis  are  presented  in  Table 
19.  Hiring  delay  accounted  for  a  statistically  signifi¬ 
cant  portion  of  the  overall  variability  in  supervisory 
ratings  of  teamwork  (F=  5.30,/  <  .05).  However, 
PROGRAM  (MnATCTC  or  FAA  Academy)  still  ac¬ 
counted  for  a  statistically  significant  portion  of  vari¬ 
ability  in  TEAMWORK  ratings  (f  =  9.89, />  <  .01),  even 
after  accounting  for  the  effects  of  hiring  delays.  This 
pattern  of  results  suggests  that  differences  in  the  mean 
supervisory  ratings  of  teamwork  for  MnATCTC  and 
FAA  Academy  graduates  were  not  merely  the  result  of 
delays  in  hiring  but  may  have  reflected  real  differ¬ 
ences  between  the  groups. 

In  contrast,  hiring  delay  was  not  a  statistically 
significant  factor  in  explaining  the  variability  of 
supervisor’s  ratings  ofTECHNlCALSKiLL(F'=  2.00,  ns). 
Differences  in  the  mean  ratings  were  attributable 
only  to  the  program  in  which  the  controller  was 
initially  trained  {F  =  4.19,  p  <  .05).  This  analysis 
suggests  that  hiring  delays  had  no  effect  on  the  mean 
ratings  of  TECHNICAL  SKILL  for  the  two  groups. 

Supervisory  ratings  of  technical  knowledge  fol¬ 
lowed  the  same  pattern  as  teamwork,  with  both 
hiring  delay  and  program  accounting  for  statistically 
significant  portions  of  the  overall  variability  in  tech¬ 
nical  knowledge  supervisory  ratings.  This  pattern  of 
results  suggested  that  differences  in  the  mean  ratings 
of  TECHNICAL  KNOWLEDGE  for  MnATCTC  and  FAA 
Academy  graduates  were  not  merely  the  result  of 
delays  in  hiring,  but  may  have  reflected  real  differ¬ 
ences  between  the  groups. 


Finally,  supervisory  ratings  of  potential  to  suc¬ 
ceed  in  the  occupation  for  the  two  groups  followed 
the  same  pattern  as  the  technical  skill  ratings. 
Hiring  DELAY  was  not  a  significant  factor  in  explain¬ 
ing  the  variability  in  ratings  of  POTENTIAL  to  succeed 
for  the  two  groups  (F=  0.02,  ns),  while  program  was 
(f  =  15.94,/  <  .001).  In  other  words,  differences  in 
supervisory  ratings  of  potential  to  succeed  in  the 
ATCS  occupation  could  not  be  attributed  to  the 
delay  in  hiring  MnATCTC  graduates. 

Joint  effects  of  acceptance  and  hiring  delay.  The 
last  analysis  of  covariance  considered  the  joint  effects 
of  the  degree  to  which  a  controller  felt  accepted  at  the 
facility  (aCCEPTANCE),  and  the  DELAY  in  hiring  experi¬ 
enced,  on  supervisory  ratings  of  TEAMWORK,  TECHNI¬ 
CAL  SKILL,  TECHNICAL  KNOWLEDGE,  and  POTENTIAL  tO 
succeed  in  the  ATCS  occupation.  The  results  of  this 
covariate  analysis  for  teamwork  supervisory  ratings 
are  presented  in  Table  20.  ACCEPTANCE  was  a  signifi¬ 
cant  ftictor  in  explaining  the  overall  variability  in 
supervisor  ratings  of  teamwork  (F=  43.04,/  ^  .001 ). 
Hiring  DELAY  was  also  a  significant  factor  {F  =  4.33, 
p  <  .05),  as  was  the  combined  effects  of  the  two 
covariates  {F  =  23.69,  p^  .01).  However,  even  after 
accounting  for  the  joint  effects  of  ACCEPTANCE  and 
hiring  DELAY,  PROGRAM  still  had  a  statistically  signifi¬ 
cant  main  effect  on  supervisor  ratings  of  teamwork 
{F=  5.27,  p<  .05).  These  results  suggest  that  differ¬ 
ences  in  teamwork  ratings  between  MnATCTC  and 
FAA  Academy  graduates  cannot  be  explained  away  as 
the  effects  of  not  being  accepted  at  the  facility  and 
long  delays  in  hiring  MnATCTC  graduates  but  may 
have  reflected  real  differences  in  the  teamwork  of 
MnATCTC  compared  with  FAA  Academy  gradu¬ 
ates. 

The  analysis  of  supervisory  ratings  of  TECHNICAL 
SKILL,  taking  into  account  the  joint  effects  of  accep¬ 
tance  and  hiring  delay,  leads  to  a  different  result 
(Table  21).  The  acceptance  covariate  was  a  signifi¬ 
cant  factor  in  the  overall  variability  of  supervisor 
ratings  ofTECHNiCAL  skill  {F=  7.83, /  <  .01)  but  not 
hiring  delay  (F  =  1.59,  ns).  After  accounting  for  the 
effects  of  the  degree  to  which  the  controller  felt 
accepted  at  the  facility,  program  (MnATCTC  or 
FAA  Academy)  accounted  for  a  statistically  signifi¬ 
cant  portion  of  the  overall  variability  in  ratings  of 
TECHNICAL  SKILL  {F  =  4.71,  /  <  .01).  These  results 
suggested  that  the  lower  ratings  on  technical  skill 
for  the  MnATCTC  graduates,  compared  with  FAA 
Academy  graduates,  could  not  be  explained  away  by 
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Table  19.  Supervisor  ratings  on  performance  dimensions,  controlling  for  delay  between  graduation  and 
reporting  to  facility 
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differences  in  the  degree  to  which  MnATCTC  and 
FAA  Academy  graduates  felt  accepted  at  the  facility 
but  may  have  reflected  real  differences  in  perfor¬ 
mance. 

The  analysis  of  covariance  for  supervisor  ratings  of 
technical  knowledge  for  the  two  groups  is  presented 
in  Table  22.  The  pattern  of  results  for  technical 
knowledge  was  again  similar  to  that  of  teamwork, 
with  PROGRAM  accounting  for  a  statistically  signifi¬ 
cant  portion  of  the  overall  variability  in  technical 
KNOWLEDGE  ratings,  even  after  accounting  for  the 
separate  and  joint  effects  of  acceptance  and  hiring 
delay.  Finally,  the  analysis  of  the  potential  to  suc¬ 
ceed  ratings  for  the  two  groups  is  presented  in  Table 
23.  The  results  for  potential  to  succeed  followed  the 
same  pattern  as  the  technical  skill  ratings,  with 
ACCEPTANCE  accounting  for  a  statistically  significant 
portion  of  the  variability  in  the  potential  to  succeed 
ratings  made  by  supervisors  {F  =  15.04,  p  <  .001). 
Hiring  delay  was  not  a  significant  factor  in  this 
analysis,  while  PROGRAM  was.  The  results  suggest  that 
the  lower  ratings  of  potential  to  succeed  given  by 
supervisors  to  MnATCTC  graduates  in  comparison 
to  FAA  Academy  graduates  could  not  be  explained  as 
the  consequence  of  differences  in  the  degree  to  which 
MnATCTC  graduates  were  accepted  in  the  facility. 
Rather,  the  supervisor  ratings  of  potential  to  suc¬ 
ceed  in  the  ATCS  occupation  may  have  reflected  real 
differences  in  performance  between  the  two  groups. 

Costs  and  benefits 

Costs 

Program  costs.  Accrued  direct  costs  for  the 
MnATCTC  program  consisted  of  a  series  of  Con¬ 
gressional  earmarks  totaling  $10.9M.  Indirect  costs 
included:  (a)  headquarters  human  resources  manage¬ 
ment  staff  time;  (b)  site  visits  by  FAA  personnel;  (c) 
CAMI  evaluation  staff  time;  (d)  evaluation  contracts; 
(e)  program  steering  committee  meetings;  (f)  re¬ 
gional  liaison  staff  time;  (g)  FAA  Academy  staff  time; 
and  (h)  Air  Traffic  staff  time.  The  total  time  and  costs 
associated  with  managing  the  CTI/ATCS  program, 
as  reported  via  e-mail,  were  prorated  across  the  five 
participating  institutions,  except  where  those  costs 
were  explicitly  attributed  to  MnATCTC  only.  A 
breakdown  of  those  annual  costs  by  fiscal  year  (FY) 
are  presented  in  Appendix  B;  the  logic  of  each  annual 


costs  worksheet  is  described  in  Table  24.  To  estimate 
the  running  cost-per-hire  for  the  MnATCTC  pro¬ 
gram,  the  cumulative  costs  each  year  were  amortized 
over  the  cumulative  actual  and  projected  number  of 
hires  through  that  year.  The  cost  per  hire  in  1 99 1  was 
high,  at  about  $281,000,  reflecting  initial  start-up 
costs  for  the  program.  This  cost-per-hire  was  reduced 
in  1992  to  about  $121,000  per  MnATCTC  graduate 
hired  by  the  FAA.  That  cost  was  further  reduced  in 
1993  to  approximately  $81,000  per  hire  as  more 
MnATCTC  graduates  entered  FAA  service.  With 
additional  hiring  in  FY94,  the  MnATCTC  cost-per- 
hire  was  stable  at  about  $57,000  through  FY95.  The 
FY96  cost  was  projected  to  be  about  $50,000  per 
graduate  with  continued  hiring. 

Cost  comparison.  Actual  and  projected  hiring  for 
MnATCTC  program  graduates  was  used  to  estimate 
the  likely  cost  per  hire  through  the  year  2003,  as 
shown  in  Figure  3.  The  costs  per  controller  at  the 
FAA  Academy  under  the  redesigned  “Train  to  Suc¬ 
ceed”  model  were  estimated  by  the  Air  Traffic  Train¬ 
ing  Work  Group  (ATTWG,  1992)  at  about  $33,000 
through  the  PV  phase.  Slightly  different  cost  figures 
can  be  obtained  from  the  FAA  Academy  Tuition 
Pricing  System  (ATPS;  FAA  Academy,  1994).  The 
per  controller  cost  for  basic  en  route  training  was 
estimated  at  about  $58,000  in  FY94  by  ATPS.  Fi¬ 
nally,  more  recent  figures,  as  coordinated  with  the 
FAA  Academy,  the  Training  Requirements  Division 
(ATZ-100),  and  Assistant  Director  of  the  Office  of 
Air  Traffic  Program  Management  (ATZ-2)  as  of 
April  1995  (Larry  Lackey,  personal  communication. 
May  1995),  estimated  the  Academy  cost  for  initial 
resident  training  at  $45,500  in  the  en  route  option^ 
The  highest  and  lowest  estimates  were  used  to  define 
a  range  for  estimated  Academy  per-controller  costs 
for  comparison  purposes. 

This  cost  analysis  suggests  that  MnATCTC  is 
competitive  with  the  FAA  Academy,  in  terms  of  the 
costs  to  produce  each  graduate  at  about  $50,000  per 
controller.  With  the  projected  hiring  of  64  graduates 
in  FY97,  and  as  many  as  100  in  FY98  and  beyond, 
MnATCTC  per  graduate  costs  may  be  lower  than  the 
FAA  Academy’s  lowest  estimated  per  controller  cost. 
Continued  hiring  of  MnATCTC  graduates  could 
reduce  the  fully  amortized  cost  per  MnATCTC  hire 
to  a  level  that  is  competitive  with  internal  FAA 


‘The  per-student  expenditure  rate  at  the  FAA  Academy  includes  incremental  costs  and  excludes  about  $13.7  million  in  capital 
costs  for  the  En  Route  program  (Sweetman,  personal  communication,  January  30,  1997). 
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Table  22.  Supervisor  ratings  of  technical  knowledge,  conlrolllng  fbr  ATCS’S  perceived  degree  of  acceptance  in  the 
faciiity  and  delay  between  graduation  and  hiring 
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Academy  costs,  despite  an  additional  $1,700,000  in 
funding  in  FY97  (P.L.  104-205).  Even  with  continu¬ 
ing  subsidies  to  the  Minnesota  Air  Traffic  Control 
Training  Program  of  $1.5  million  per  year,  as  re¬ 
quested  in  congressional  testimony  by  the  Director  of 
that  program  (Pointer,  March  1995),  the  cost  per 
MnATCTC  hire  is  likely  to  be  very  competitive  with 
FAA  Academy  per  controller  costs  in  the  en  route 
option.  Elimination  of  continued  subsidies  to  the 
MnATCTC  program  in  fiscal  year  1997  and  beyond, 
as  recommended  in  the  1993  National  Performance 
Review  (p.  98)  would  only  increase  the  cost  advan¬ 
tage  of  the  MnATCTC  program  over  the  FAA  Acad¬ 
emy  in  providing  initial  technical  training  for  the  cn 
route  option  over  the  long  term. 

Benefits 

Program  benefits 

A  cost  advantage,  however,  does  not  necessarily 
result  in  benefits  to  the  taxpayer.  Therefore,  the  next 
step  was  to  analyze  benefits  accruing  from  the 
MnATCTC  program  relative  to  costs.  Four  classes  of 
program  benefits  that  could  be  clearly  expressed  in 
dollars  were  identified:  (a)  avoided  screening  costs; 
(b)  avoided  Academy  training  costs;  (c)  avoided  PV 
costs;  and  (d)  savings  from  reduced  time  to  FPL. 
Screening  costs  refers  to  the  agency  costs  in  adminis¬ 
tering  either  the  former  FAA  Academy  ATCS 
Nonradar  Screen  program  (Broach  &  Manning,  1994) 
through  March  1992,  or  the  replacement  computer¬ 
ized  ATCS  Pre-Training  Screen  (Broach  &  Brecht- 
Clark,  1994)  from  June  1992  through  June  1996. 
The  costs  of  the  former  ATCS  Nonradar  Screen  (FAA 
Academy  course  50321)  were  estimated  at  about 
$10,000  per  controller  in  1991.  The  replacement 
computerized  test  battery  cost  about  $1,500  per 
examinee  (Broach  &  Brecht-Clark,  1994).  The  FAA 
avoided  incurring  these  costs  for  MnATCTC  gradu¬ 
ates  as  they  did  not  go  through  the  Nonradar  Screen 
or  the  ATCS  Pre-Training  Screen. 

Avoided  Academy  training  costs  refers  to  costs 
incurred  by  the  agency  under  the  redesigned  Acad¬ 
emy  program  of  about  $45,500  per  controller  in  the 
en  route  option.  The  redesigned  FAA  Academy  pro¬ 
gram  will  consist  of  three  phases  or  modules:  academ¬ 
ics  (Phase  I);  techniques  (Phase  II);  and  skills  building 
(Phase  III;  Air  Traffic  Training  Group,  1996).  There 
were  no  plans  for  MnATCTC  graduates  to  attend  any 


phase  of  the  FAA  Academy  ATCS  training  program, 
as  of  1995,  thereby  saving  the  agency  approximately 
$45,500  per  year  per  graduate  hired. 

Performance  verification  (PV)  is  the  Air  Traffic 
evaluation  of  the  readiness  of  a  trainee  to  enter  the 
field  for  on-the-job  training.  PV  was  instituted  in 
June  1992  as  part  of  the  overhaul  of  the  ATCS 
curriculum  by  the  ATTWG  <1992).  The  core  of  PV 
requires  bringing  experienced  field  controllers  in  as 
evaluators  for  each  trainee.  The  costs  of  PV  based  on 
<a)  five  days  salary  for  a  FG-14  FPL  controller,  (b) 
five  days  per  diem  in  Oldahoma  City,  and  (c)  an 
average  round-trip  fare  of  $250.  The  rounded  sum  of 
these  costs  was  prorated  across  four  examinees  to 
arrive  at  a  cost  estimate  of  about  $500  per  PV  exam¬ 
inee.  From  1992  dirough  the  present,  the  agency 
delegated  the  PV  function  to  the  schools  under  the 
supervision  and  direction  of  the  Air  Traffic  Perfor¬ 
mance  Verification  Division  <ATZ-400).  Thus,  the 
agency  has  avoided  those  PV  costs  between  1992  and 
the  present.  There  are  no  plans  for  MnATCTC  gradu¬ 
ates  to  undergo  PV  at  the  FAA  Academy,  and  the  FAA 
will  continue  to  avoid  PV  costs  for  these  controllers. 

The  final  category  of  benefit  to  the  agency  is  in  the 
reduction  of  the  time  to  FPL.  The  1991  comprehen¬ 
sive  review  of  ATCS  training  estimated  en  route  field 
training  costs  over  a  36  month  average  time  to  FPL  at 
$131,739  (ATTWG,  1991a).  This  total  cost  was 
prorated  over  the  three  years  to  provide  an  estimate  of 
the  annual  OJT  cost  of  about  $43,000.  Reductions  in 
time  to  FPL  reduce  the  amount  spent  on  training, 
which  represents  a  savings  to  the  agency. 

Cost-benefit  analysis 

The  cost-benefit  analysis  focused  on  determining 
the  point  at  which  cumulative  program  benefits  would 
balance  or  exceed  cumulative  MnATCTC  program 
costs.  Costs  were  calculated  from  1991  through  the 
year  2003  as  the  sum,  each  year,  of  identified  direct 
and  indirect  costs,  as  shown  in  each  fiscal  year’s 
worksheet  in  Appendix  B  and  Table  24.  Benefits 
accrued  from  avoided  costs  were  calculated  for  each 
year  by  multiplying  the  number  of  MnATCTC  gradu¬ 
ates  hired  by  the  avoided  screening,  three  phases  of 
initial  qualification  training  at  the  FAA  Academy, 
and  PV  costs,  as  shown  in  the  cost-benefit  analysis 
worksheets  in  Appendix  B.  The  logic  of  the  cost- 
benefit  worksheet  is  presented  in  Table  25.  The 
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reduction  in  time  to  FPL,  while  not  statistically 
significant,  was  multiplied  by  the  average  OJT  cost 
per  year  to  assess  the  financial  utility  of  even  small 
gains  in  efficiency;  that  product  was  multiplied  by 
the  number  of  graduates  per  year  to  provide  a  rough 
estimate  of  savings  attributable  to  reductions  in 
time-to-FPL.  Avoided  costs  and  OJT  savings  were 
then  summed. 

The  cumulative  costs  and  benefits  were  then  com¬ 
pared,  as  shown  in  Figure  4,  to  identify  the  probable 
time  frame  in  which  benefits  accruing  from  the 
MnATCTC  program  might  balance  or  exceed  cumu¬ 
lative  costs.  Without  considering  other  factors,  such 
as  the  field  training  resource  implications  of  the 
differing  skill  levels  indicated  by  the  ratings  data,  it 
appears  that  cumulative  benefits  to  the  FAA  from  the 
MnATCTC  program  might  outweigh  sunk  costs  by 
about  FY1998,  based  on  the  projected  hiring  rates. 
The  ratio  of  cumulative  benefits  to  costs  is  illustrated 
in  Figure  5.  Overall,  the  MnATCTC  program  might 
return  at  least  one  dollar  in  avoided  costs  and  savings 
for  every  dollar  invested  by  FYl  998,  based  on  current 
projected  hiring  rates. 


DISCUSSION 

Overall,  the  CTI/ATCS  program  appears  to  be 
meeting  its  operational  objectives  in  terms  of  em¬ 
ployee  diversity,  progress  in  field  training,  and  con¬ 
troller  performance.  The  MnATCTC  program  appears 
to  be  providing  a  greater  proportion  of  women  to  the 
field  facilities  than  has  been  provided  through  the 
FAA  Academy.  However,  the  MnATCTC  program 
does  not  appear  to  be  a  better  source  for  minority 
controllers  than  existing  workforce  sources  such  as 
the  FAA  Academy.  There  are  few  significant  differ¬ 
ences  between  MnATCTC  and  FAA  Academy  gradu¬ 
ates  in  terms  of  training  measures  after  taking  into 
account  inter-facility  differences  in  programs. 
MnATCTC  graduates  appear  to  do  better  than  FAA 
Academy  graduates  on  some  training  measures,  and 
FAA  Academy  graduates  do  better  on  others.  The  net 
effect  appears  to  be  that,  overall,  MnATCTC  gradu¬ 
ates  require  about  the  same  amount  of  days  and  hours 
of  OJT  as  FAA  Academy  graduates.  The  attrition 
rates  for  the  two  groups  are  similar.  A  relatively  small 
number  of  MnATCTC  graduates  had  been  certified 
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as  FPL  controllers  as  of  June  1995,  compared  with  a 
larger  number  of  FAA  Academy  graduates.  However, 
the  times  required  for  the  two  groups  to  certify  were 
about  the  same.  Moreover,  the  accession  analysis 
suggested  that  the  number  of  MnATCTC  graduates 
that  could  be  reasonably  expected  to  achieve  FPL, 
and  their  times  to  FPL,  are  likely  to  be  comparable  to 
those  for  FAA  Academy  graduates. 

On  the  other  hand,  the  ratings  data  suggest  some 
caution  in  concluding  that  the  MnATCTC  is  pro¬ 
ducing  graduates  that  are  strictly  comparable  to  gradu¬ 
ates  of  the  FAA  Academy  —  in  terms  of  their 
teamwork,  technical  skill,  knowledge,  and  potential 
to  succeed.  Overall,  the  mean  ratings  of  MnATCTC 
graduate  teamwork,  technical  skill,  knowledge,  and 
potential  to  succeed  were  statistically  lower  than  the 
mean  ratings  of  the  Academy  comparison  group, 
even  after  taking  into  account  other  factors,  such  as 
the  degree  to  which  graduates  felt  accepted  at  the 
facility  and  hiring  delays.  Alternative  explanations 
for  these  difference  might  include  discomfort  with 
MnATCTC  graduates  and  rating  errors  associated 
with  stringency  and  leniency.  Another  explanation 
might  be  that  FAA  ATCS  supervisors  are  as  yet 
uncomfortable  with  controllers  entering  the  work 
force  through  other  than  the  traditional  pipelines. 
Or,  it  maybe  that  the  differences  in  ratings  reflect  real 


differences  in  performance.  The  development  of  ob¬ 
jective  measures  of  the  core  technical  performance  of 
controllers,  such  as  envisioned  for  the  Separation  and 
Control  Hiring  Assessment  (SACHA)  procurement 
(FAA,  1991)  and  its  successor,  the  Air  Traffic  Selec¬ 
tion  and  Training  (AT-SAT)  program  (FAA,  1996), 
may  provide  better  assessment  tools  in  future  program 
evaluations  than  subjective  ratings  that  may  be  influ¬ 
enced  by  stringency,  leniency,  and  other  rating  errors. 

Finally,  the  cost-benefit  data  suggest  that  benefits 
may  accrue  to  the  agency  by  using  MnATCTC  as  an 
alternative  workforce  recruiting  and  training  source 
for  the  en  route  option.  It  appears  from  this  analysis 
that  MnATCTC  can  produce  graduates  at  a  cost  that 
is  competitive  with  the  FAA  Academy,  even  with 
continuing,  congressionally-mandated  financial  sup¬ 
port  from  the  FAA  for  the  Minnesota  program.  The 
cost-benefit  analysis  for  MnATCTC  also  suggests 
that,  given  current  hiring  projections,  the  MnATCTC 
will  have  a  positive  return-on-investment  by  about 
FY1998.  The  majority  of  that  benefit  accrues  in  the 
form  of  avoided  training  costs  at  the  FAA  Academy. 
However,  this  analysis  does  not  consider  other  costs 
to  the  agency.  For  example,  the  differences  in  skill 
level  for  MnATCTC  graduates  may  place  greater 
burdens  over  time  on  facility  training  resources  to 
bring  those  graduates  up  to  a  common  standard  of 
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Figure  5.  MnATCTC  cumulative  benefit-to-cost  ratio 
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performance.  A  significant  effort  will  be  required  of 
the  FAA  to  recruit  minority  candidates  for  the  ATCS 
occupation  in  view  of  the  difficulties  experienced  by 
the  MnATCTC  to  recruit  a  diverse  student  popula¬ 
tion.  Maintaining  the  technological  currency  of  the 
MnATCTC  program  on  the  NAS  architecture  as  it 
evolves  may  impose  another  cost  on  the  agency  that 
is  not  reflected  in  this  cost-benefit  analysis.  More¬ 
over,  improvements  in  efficiency  at  the  FAA  Acad¬ 
emy,  reducing  agency  costs,  would  reduce  the  apparent 
financial  benefits  of  the  MnATCTC  program. 

In  conclusion,  this  first  summative  evaluation 
found,  on  one  hand,  that  the  MnATCTC  program, 
under  the  Collegiate  Training  Initiative  for  Air  Traf¬ 
fic  Control  Specialists  (CTI/ATCS)  umbrella,  ap¬ 
pears  to  be  meeting  defined  program  objectives,  in 
terms  of  recruiting  women  into  this  traditionally 
male  occupation.  However,  MnATCTC  has  been 
less  successful  than  the  FAA  in  recruiting  minorities 
into  the  ATCS  occupation,  despite  a  substantial 
investment  in  a  national  recruiting  program. 
MnATCTC  graduate  progress  through  the  controller 
field  training  appears  to  be  on  a  par  with  that  of 
persons  entering  the  occupation  through  the  FAA 
Academy,  based  on  the  objective  tracking  and  attri¬ 
tion  data.  However,  the  subjective  ratings  are  less 
supportive,  with  MnATCTC  graduates  having  lower 
average  ratings  of  teamwork,  technical  skill,  technical 
knowledge,  and  potential  to  succeed  in  the  ATCS 
occupation  than  Academy  graduates,  even  after  con¬ 
trolling  for  the  fact  that  MnATCTC  graduates  felt 
less  well  accepted  in  the  facility  and  experienced 
significant  delays  between  graduation  and  hiring. 
These  differences  may  reflect  actual  performance 
differences,  or  perhaps,  discomfort  with  persons  en¬ 
tering  the  occupation  through  other  than  traditional 
routes.  Research  on  supervisory  attitudes  and  expec¬ 
tations  of  new  controllers  might  provide  a  basis  for 
understanding  these  differences  in  ratings  and  for 
designing  training  management  interventions  and 
strategies  to  mitigate  any  discomfort  and  ease  the 
organizational  socialization  process  for  new  control¬ 
lers  in  field  facilities.  Differences  in  personality  and 
biographical  background  between  persons  that  enter 
through  the  collegiate  and  competitive  channels  might 
also  be  investigated  as  potential  explanations  for 
differences  in  performance.  Objective  measures  of 
core  technical  performance,  currently  under  develop¬ 
ment,  may  be  used  in  future  evaluations  to  provide  a 
more  definitive  comparison  of  the  safety  and  efficiency 


of  CTI/ATCS  graduates  to  that  of  controllers  who 
entered  the  occupation  through  the  FAA  Academy. 
Finally,  the  cost-benefit  analysis  suggests  some  long¬ 
term  benefit  for  the  FAA  by  utilizing  the  MnATCTC 
program  as  an  alternative  workforce  source.  Overall, 
the  pattern  of  results  in  this  first  summative  evalua¬ 
tion  suggests  that  expansion  of  the  CTI/ATCS  pro¬ 
gram  to  additional  educational  institutions  might  be 
considered  by  the  FAA  as  part  of  an  overall  strategy 
for  staffing  the  National  Airspace  System. 

REFERENCES 

Air  Traffic  Control  Specialist  Hiring  &  Training 
Workgroup  (1996).  Air  traffic  control  specialist 
hiring  and  training  workgroup  results  and  recom¬ 
mendations.  Washington,  DC;  Federal  Aviation 
Administration  Office  of  Air  Traffic  Program 
Management  Career  Systems  Division  (ATZ- 
200). 

Air  Traffic  Training  Workgroup  (1991a).  ATTWG 
Training  plan:  Overview.  Washington,  DC:  Fed¬ 
eral  Aviation  Administration  Office  of  AirTraffic 
Program  Management  Training  Requirements 
Division  (ATZ-100). 

Air  Traffic  Training  Workgroup.  (1991b).  Controller 
resource  management  conference:  Summary  of  the 
meeting.  Washington,  DC;  Federal  Aviation  Ad¬ 
ministration  Office  of  Air  Traffic  Program  Man¬ 
agement  Training  Requirements  Division  (ATZ- 
100). 

Bloom,  B.  (1967).  Toward  a  theory  of  testing  which 
includes  measurement-evaluation-assessment. 
(CSEIP  Occasional  Report  No.  9).  Los  Angeles, 
CA:  University  of  California  at  Los  Angeles  Cen¬ 
ter  for  the  Study  of  Evaluation  of  Instructional 
Programs. 

Borman,  W.C.,  White,  L.A.,  Pulakos,  E.D.,  &  Oppler, 
S.H.  (1991).  Models  of  supervisory  job  perfor¬ 
mance  ratings.  Journal  of  Applied  Psychology,  76, 
863-72. 

Broach,  D.  &  Brecht-Clark,  J.  (1994).  Validation  of  the 
Federal  Aviation  Administration  air  traffic  control 
specialist  Pre-Training  Screen.  (DOT/FAA/AM- 
94/4).  Washington,  DC:  Federal  Aviation  Ad¬ 
ministration  Office  of  Aviation  Medicine.  (NTIS 
No.  ADA277549) 


40 


Broach,  D.  &:  Manning,  C.  A.  (1994).  Validity  of  the  air 
traffic  control  specialist  Nonradar  Screen  as  a predic¬ 
tor  of  performance  in  radar-hosed  air  traffic  control 
training,  (DOT/FAA/AM-94/9).  Washington, 
DC:  Federal  Aviation  Administration  Office  of 
Aviation  Medicine.  (NTIS  No.  ADA279745) 

Campbell,  J.P.  (1990).  Modeling  the  performance  pre¬ 
diction  problem  in  industrial  and  organizational 
psychology.  In  M.D.  Dunnctte  &  L.M.  Hough 
(Eds.),  Handbook  of  industrial  and  organizational 
psychology.  Volume  2  (2nd  Ed.),  pp.  687  -  732. 
Vslo  Alto,  CA:  Consulting  Psychologist  Press. 

Campbell,  |.  &  Cook,  D.T.  {1976).  The  design  and 
conduct  of  quasi-experiments  and  true  experi¬ 
ments  in  field  settings.  In  M.D.  Dunnette  {Ed  ), 
Handbook  of  industrial  and  organizational psychol- 
ogy  iipp.  223  -  326).  Chicago,  IL:  Rand  McNally. 

Campbell,  J.P.,  McCIoy,  R.A.,  Oppler,  S.H.,  &  Sager, 
C.E.  {1993).  A  theory  of  performance.  In  N. 
Schmitt,  W.C.  Borman  and  Associates  (Eds.)^ 
P’ersonnd  selection  in  organizations  (pp.  35  •“  70). 
San  Francisco:  Jossey-Bass. 

Department  ofTransportatton.  (1993,  May).  Report  on 
the  audit  of  Federal  Aviation  Administration  spon¬ 
sored  higher  education  programs.  (DOT/OIG  Re¬ 
port  AV-FA-010)- Washington,  DC:  Department 
of  Transportation,  Office  of  the  Inspector  Generd. 

FAA  Academy.  (1994,  November).  Academy  Tuition 
Pricing  System.  Oklahoma  City,  OK:  Program 
and  Project  Management  Division  (AMA-100). 

Federal  Aviation  Administration.  (1984).  Employment 
program  for  developmental  air  traffic  control  spe¬ 
cialists.  (FAA  Order  3330. 30C,  September  27, 
1984).  Washington,  DC:  Author. 

Federal  Aviation  Administration.  (1985).  National  air 
traffic  training  program  tracking  system.  (FAA  Or¬ 
der  3120.22A,  July  1,  1985).  Washington,  DC: 
Author. 

Federal  Aviation  Administration.  (1988).  National  air 
traffic  training  program  en  route  instructional  pro¬ 
gram  guide.  (EP  12-0-1  E).  Oklahoma  City,  OK: 
Federal  Aviation  Administration  Academy. 

Federal  Aviation  Administration.  (1993).  Air  traffic 
control  specialist  on-the-job  training  and  position 
certification.  (FAA  Order  3120.24A,  Change  1, 
August  30,  1993).  Washington,  DC:  Author. 


General  Accounting  Office(1989a).  Con¬ 

tinued  improvements  needed  in  FAAs  controller 
field  training  program.  (GAO/RCED-89-93). 
Washington,  DC:  Author. 

General  Accounting  Office  (1989b).  Aviation  safety: 
Serious  problems  continue  to  trouble  the  air  traffic 
control  work  force.  (GAO/RCED-89-1 12).  Wash¬ 
ington,  DC:  Author. 

E.L.  Hamm  &  Associates  Inc.,  &  Hampton  University. 
(1990,  July).  Preliminary  draft  of  the  research  plan 
for  the  air  traffic  control prototype  curriculum,  (In¬ 
terim  report  submitt^  under  FAA  contract 
DTTAOi-90-Y-OlO34)..  Washington,  DC:  Fed¬ 
eral  Aviation  Administration  Office  of  Training 
and  Higher  Education. 

Hartel,  C.E.J.,  &  Hartel,  G.F.  (1995).  Controller  re¬ 
source  management  —  What  can  we  learn  from 
aircrews?  (DOT/FAA/AM-95/21).  Wadiington, 
DC:  Federal  Aviation  Administration  Office  of 
Aviation  Medicine.  (NT£S  No.  ADA297386) 

Hoffman,  C.C.,  Nathan,  B.R.,  &Holden,L.M.  (1991). 
A  comparison  of  validauon  criteria:  CSbjective 
versus  subjective  performance  measures  and  self- 
versus  supervisor  ratings.  Personnel PsycholoQi,  44, 
601  - 18. 

Manning,  C.A.  (1990).  Attrition  from  ATCSfieldtrain- 
ing.  [Unpublished  manuscript].  Oklahoma  City, 
OK:  Federal  Aviation  Administration  Civil  Aero- 
medical  Institute. 

Manning,  C.A.,  Della  Rocco,  P.S.,  &  Bryant,  K.D. 
(1989).  Prediction  of  success  in  FAA  air  traffic 
control  field  training  as  a  function  of  selection  and 
screening  test  performance.  (DOT/FAA/AM-89/ 
6).  Washington,  DC:  Federal  Aviation  Adminis¬ 
tration  Office  of  Aviation  Medicine.  (NTIS  No. 
ADA209327) 

Means,  B.,  Mumaw,  R.  Roth,  C.,  Schlager,  M, 
McWilliams,  E.,  Gagne,  E.,  Rice,  V.,  Rosenthal, 
D.,  &  Heon,  S.  (1988).  ATC  training  analysis 
study:  Design  of  the  next-generation  ATC  training 
system.  (Final  report  delivered  under  OPM  work 
order  342-036).  Alexandria,  VA:  HumRRO  In¬ 
ternational,  Inc. 

Morrison,  J.E., Fotohui,  C.H.,  &Broach, D.  (1996). rf 
formative  evaluation  of  the  Collegiate  Training  Ini¬ 
tiative  -Air  Traffic  Control  Specialist  ( CTI-A  TCS) 
program.  (DOT/FAA/AM-96/6),  Washington, 
DC:  Federal  Aviation  Administration  Office  of 
Aviation  Medicine.  (NTIS  No.  ADA305307) 


41 


National  Performance  Review.  (1993,  September)- 

red  tape  to  results:  Creating  a  government  that  works 
better  and  costs  less.  Washington,  DC;  U.S.  Gov¬ 
ernment  Printing  Office. 

Newcomb,  L.C.,  &  Jerome,  G.C.  (1994,  April).  CRM 
techniques  applied  to  pilot-controller  interfiice. 
Safe  application  of  technology  in  corporate  aviation: 
Proceeding  of  the  39th  annual  Corporate  Aviation 
Safety  Seminar  (CASS),  pp.  1 15  -  25.  Arlington, 
VA:  Flight  Safety  Foundation. 

Northern  NEF,  Inc.  (1988).  Flight  plan  far  the  future: 
Training  controllers  for  the  national  air  traffc  as¬ 
tern.  (Final  report  delivered  under  OPM  subcon¬ 
tract  agreement  SubOPM88-04).  Colorado 
Springs,  CO:  Author. 

Norusis,  M.J.  (1990).  SPSS  Advanced  statistics  user’s 
guide.  Chicago,  IL;  SPSS,  Inc. 

Office  of  Deputy  Associate  Administrator  for  Appraisal. 
(1992,  September).  An  assessment  of  contractor- 
provided  en  route  ATC  training.  (Report  89-2). 
Washington,  DC:  Federal  Aviation  Administra¬ 
tion  Office  of  the  Deputy  Associate  Administra¬ 
tor  for  Appraisal. 

Office  of  Training  and  Higher  Education  ( 1 989) .  Flight 
Plan  far  Training.  Washington,  DC:  Federal  Avia¬ 
tion  Administration  Associate  Administrator  for 
Human  Resources  Management. 

Performance  Verification  Division.  (Undated).  Perfor¬ 
mance  verfacation:  TDY trainingguide.  Oklahoma 
City,  OK;  Federal  Aviation  Administration  Of¬ 
fice  of  Air  Traffic  Program  Mani^ement  Perfor¬ 
mance  Verification  Division  (ATZ-400). 

Rossi,  P.,  &  Freeman,  H.  (1985).  Evaluation:  A  system¬ 
atic  approach.  (3rd  Ed.).  Beverly  Hills,  CA:  Sage. 


Schultz,  S.R.,  &  Marshall-Mies,  J.  (1988).  FAAATCS 
recruitment:  Assessment  of  current  procedures  and 
development  of  recommendations  for  an  integrated 
program.  (HumRRO  International,  Inc.  Techni¬ 
cal  Report  88-05  produced  under  OPM  work 
order  342-53).  Alexandria,  VA:  Human  Re¬ 
sources  Research  Organization  (HumRRO)  In¬ 
ternational  Inc. 

Seamster,T.L.,  Cannon,  J.R.,  Pierce,  R.M.,  &  Redding, 
R.E.  (1992).  Analysis  of  en  route  air  traffic  con¬ 
troller  team  communication  and  controller  re¬ 
source  management  (CRM).  Proceedings  of  the 
Human  Factors  Society  36th  Annual  Meeting,  pp. 
66  -  70.  Santa  Monica,  CA:  Human  Factors  and 
Ergonomics  Society. 

Sherman,  P.J.,&Helmreich,R.L.  (1993,  January).  The 
controller  resource  management  attitudes  question¬ 
naire  (CRMAC)):  Review,  methodology,  and  results. 
(NASA/UT/FAATechnical  Report  93-1).  Wash¬ 
ington,  DC;  Federal  Aviation  Administration 
Office  of  Aviation  Research  Air  Traffic  Human 
Factors  Program. 

Smith,  P.C.  (1976).  The  problem  of  criteria.  In  M.D. 
Dunnette  (Ed.),  Handbook  of  industrial/organiza¬ 
tional  psychology  (pp.  745  -  75).  Chicago,  IL: 
Rand  McNally. 

Stark,  D.W.  (1994).  The  validity  of  cognitive  and  non- 
cognitive  predictors  over  time.  Unpublished  doc¬ 
toral  dissertation,  the  University  of  Tulsa. 

Thompson,  M.S.  (1980).  Bentfit-cost  analysis  for  pro¬ 
gram  evaluation.  Beverly  Hills,  CA;  Sage. 

U.S.  Congress.  (1976,  January).  House  committee  on 
government  operations  recommendations  on  air  traffic 
control  training.  Washington,  DC:  Author. 


42 


A{q[)ei^ix  A 

Sample  Peiformanee  Rating  Packets 
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Date 


[OJT  Instructor] 

[Facility] 

[Mailing  address] 

[City,  ST,  Zip-xxxx] 

Dear  {OJT  Instructor] 

The  Collegiate  Training  Initiative  for  Air  Traffic  Control  Specialists  (CTI)  program  was  implemented  by  the  agency 
in  1990  in  order  to  assess  the  feasibility  of  developing  and  implementing  initial,  entry-level  ATCS  technical  training 
at  selected  educational  institutions  throughout  the  country.  [Controller]  graduated  from  one  of  die  five  participating 
CTI  schools,  and  was  placed  directly  into  a  field  facility  without  going  through  the  initial  fraining  at  the  FAA 
Academy.  This  survey  is  part  of  a  series  of  studies  being  conducted  by  the  Civil  Aeromedical  Institute  (CAMI)  to 
evaluate  the  feasibility  of  the  CTI  program.  These  survey  results  will  be  used  to  assess  how  well  the  CTI  programs 
have  trained  their  graduates  relative  to  FAA-trained  developmentals. 

In  this  evaluation,  supervisors  of  and  selected  OJT  instructors  for  die  CTI  graduates  currently  employed  by  die  FAA 
will  receive  this  CTI  Controller  Profile.  In  the  survey,  you  are  asked  to  assess  the  performance  of  [controller] —  his 
or  her  technical  skills,  technical  knowledge,  and  teamwork  —  relative  to  all  other  controllers  you  have  known  at  the 
same  point  in  their  career. 

The  same  survey  is  being  sent  to  die  supervisors  of  and  OJT  instructors  for  a  comparison  group  of  FAA  Academy- 
trained  controllers,  in  order  to  provide  a  basis  for  comparing  how  well  the  CTI  and  Academy  programs  frained  new 
controllers  for  success  in  field  training.  Similar  surveys  are  also  being  sent  to  the  controllers  to  obtain  their  self- 
assessments  as  well.  The  results  of  the  self-,  supervisor-,  and  instructor-assessments  will  be  combined  for  each 
person,  and  then  aggregated  within  groups  for  analysis.  Let  us  stress  to  you  that  the  focus  of  the  study  is  on 
evaluating  the  CTI  and  Academy  programs. 

In  other  words,  only  group  statistics  shall  be  used  in  any  reports.  Data  about  individuals  are  confidential,  and  shall 
not  be  released.  The  data  collected  in  this  study  and  its  results  shall  not  be  used  in  any  way,  by  any  member, 
employee,  representative,  or  contractor,  of  the  agency  to  effect  your  assignments,  training,  working  conditions,  or 
status.  The  surveys  shall  not  be  retained,  recorded,  or  copied  in  any  way  at  the  facility  for  any  purpose. 

Finally,  the  profile  data  from  this  survey  are  vital  to  evaluating  the  CTI  program  for  the  agency,  the  Department  of 
Transportation,  and  the  CTI  schools.  We  strongly  urge  you  to  complete  the  survey  and  return  it  in  the  postage  paid 
envelope  as  soon  as  possible. 

/s/ 

[Name] 

[Air  Traffic  title] 
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Collegiate  Training  Initiative  (CTI)  Graduate  Profile 


PURPOSE  OF  SURVEY 


This  survey  is  part  of  a  series  of  studies  to  evaluate  the  Collegiate  Training  Initiative  (CTI). 
Your  first-hand  impressions  of  the  performance  of  [controller  (SSN)]  are  very  important  to 
determining  if  the  CTI  program  is  accomplishing  the  goal  of  putting  high-aptitude  persons  into 
the  field. 


BACKGROUND  OF  SURVEY 

The  CTI  program  ivas  initially  implemented  in  1990.  [controller]  graduated  from  one  of  the  5 
participating  institutions.  The  purpose  of  this  followup  study  is  to  assess  the  performance  and 
progress  of  [controller]  as  part  of  the  overall  evaluation  of  the  CTI. 

THANKS  FOR  YOUR  COOPERATION 

The  profile  data  from  this  survey  are  vital  to  evaluating  the  CTI  for  the  agency.  Your 
cooperation  and  thoughtful  consideration  of  [controller] ’s  skills,  teamwork,  and  knowledge  is 
greatly  appreciated  —  and  needed. 

A  FINAL  WORD 

The  researchers  at  CAMI  know  that  it  seems  like  there  a  lot  of  surveys  in  the  FAA.  This  survey, 
however,  has  been  mailed  to  a  very  specific  group,  as  opposed  to  a  more  general  sample  like  the 
Job  Satisfaction  Survey.  We  also  know  that,  often,  not  much  seems  to  ever  be  heard  about  the 
results  of  a  survey.  To  combat  this  perception,  we’d  like  to  send  a  short  summary  of  the  results  to 
you,  when  they  become  available.  Just  fill  out  a  mailing  label  or  envelope,  and  include  it  in  the 
return  envelope. 

THANK  YOU  VERY  MUCH! 
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Please  remember  that  this  survey: 

•  Is  for  research  purposes  only  under  5  USC  1301, 2301,  &  3304; 

•  Shall  be  used  only  for  the  evaluation  of  the  CTI  by  CAMI  researchers; 

•  Is  confidential  and  only  group  statistics  shall  be  used  in  any  reports; 

•  Shall  not  be  discussed  with  the  controller  or  any  other  person; 

•  Shall  not  effect  assignments,  working  conditions,  or  status  of  the  controller;  and 

•  Shall  not  be  copied,  recorded,  or  retained  at  the  facility  for  any  purpose. 

TEAMWORK:  Consider  the  controller’s  teamwork,  relative  to  all  other  controllers  you  have  observed  at  the 
same  point  in  training.  Use  the  scale  below  to  profile  [controller]’s  teamwork. 


iiiiiil 

®  =  Very  well 

Iiiiiil 

®  =  Marginally 

\ 

1  @  SB  CanH  say,  donH  know,  or  not  dpplktdfle 

Relative  to  all  other  controllers  you  have  observed  at  the  same 

point  in  their  career,  how  well  does  this  controller: 

k--' . 

■ 

1. 

Work  in  a  team . 

'0) 

0 

2. 

Engage  other  team  members  in  solving  a  problem . 

[0 

0 

1® 

3. 

Lead  the  team  in  solving  problems,  making  improvements,  etc. 

® 

0 

0 

'■® 

4. 

Earn  the  respect  of  team  members . 

0 

0 

\0\ 

5. 

Build  camadarie  or  spirit  appropriately  within  the  team . . 

0 

0 

6. 

Discourage  horse  play  or  other  disruptive  behavior . 

0 

0 

7. 

Support  or  aid  team  members  in  stressful  situations . 

0 

0 

8. 

Maintain  awareness  of  own  ability  limits . 

® 

® 

0 

,® 

9. 

Accept  feedback  regarding  performance . 

® 

0\ 

0 

10. 

Seek  additional  information  when  confronted  with  a  problem 

® 

0 

0 

11. 

Evaluate  alternative  solutions  to  a  problem . 

® 

0 

0\ 

0 

12. 

Perform  confidently .  i 

® 

1 

0 

0\ 

® 

13. 

Perform  consistently  day  after  day  regardless  of  circumstances 

® 

0 

0 

® 

14. 

Adapt  to  changing  conditions  or  circumstances  on  the  job . 

,® 

® 

0 

0 

® 

15. 

Tolerate  stressful  situations . 

® 

■0 

0 

@ 
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1  ®  =  Strongly 

■= 

(D  =  Very  much 

k 

/ 

®  =  Not  very  much 

i 

? 

1 

1;  ®  =  C<m’t  say.  don’t  know,  or  not  applicable 

i  ■ 

1'  ‘ 

1 

Relative  to  all  other  controllers  you  have  observed  at  this  point  in 

F 

r 

their  career,  to  what  degree  do  you  believe  that  this  controller: 

Tc  nart  nf  tViP  tP.Jim  . 

(S) 

© 

© 

© 

Might  be  more  accepted  in  another  work  group  or  team . 

® 

© 

© 

© 

Fits  in  with  his  or  her  current  work  group  or  team . 

H 

© 

w 

© 

© 

Is  treated  fairly  by  her  or  his  current  work  group  or  team . 

© 

(2) 

© 

© 

Is  affected  negatively  (with  regard  to  work  performance)  by  how 

the  other  controllers  on  his  or  her  current  team  view  her/him . 

:® 

© 

© 

© 

© 

SKILLS:  Now  consider  the  controller’s  technical  skill  in  performing  air  traffic  control  tasks,  relative 
to  all  other  controllers  you  have  observed  at  the  same  point  in  their  career.  Use  the  scale  below  to 
profile  [controller] ’s  technical  skills^ _ _ 


®  Exeeeptionally 

©  =  Very  well 

i  O  ts  Acceptably 

©  =  Marginally 

s  Can’t  say.  don’t  know,  or  not  applicable 

Relative  to  all  other  controllers  you  have  observed  at  this  point  in 
their  career,  how  well  does  this  controller: 

21. 

Ensure  separation  using  vectors,  speed,  &  altitude . 

© 

© 

© 

© 

22. 

Maintain  an  orderly  flow  of  traffic . 

® 

© 

© 

© 

© 

23. 

Sequence  traffic . 

® 

© 

© 

© 

© 

24. 

Perform  pointouts  and  handoffs . 

:® 

© 

© 

© 

© 

25. 

Manage  (surface)  traffic  movement  areas . 

® 

© 

fi' 

© 

© 

26. 

Recover  from  and  correct  errors,  slips,  and/or  mistakes . 

® 

© 

’© 

© 

^© 

27. 

Prioritize  actions . 

® 

© 

© 

© 

■© 

28. 

Maintain  situational  awareness . 

® 

© 

© 

© 

© 

f. 

29. 

Issue  clearances  using  appropriate  phraseology . 

® 

© 

© 

© 

"© 

30. 

Conduct  relief  briefing . 

® 

© 

© 

© 

© 

31. 

Post  flight  data  on  flight  progress  strips . 

® 

© 

© 

© 

© 
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KNOWLEDGE:  Finally,  consider  the  controller’s  technical  knowledge  about  air  toffic  control, 
relative  to  all  other  controllers  you  have  observed  at  the  same  point  in  training.  Use  the  scale  below  to 
profile  [controller] ’s  technical  knowledge: 


Most  everything 

Iliiii 

®  =  Much 

^^xSome 

©  =  Very  little 

iM«^Cori^tsoy^don*tknow^ormtopplicobU 

Relative  to  all  other  controllers  you  have  observed  at  the  same 

1 

^  0- 

point  in  their  career,  how  much  does  this  controller  know  about: 

> 

32. 

Airspace  configuration  in  sector  and/or  area  of  specialization 

© 

© 

© 

© 

33. 

Traps,  hot  spots,  and  traffic  patterns  or  flows  in  the  sector/area 

© 

© 

© 

1®^ 

34. 

Relevant  sector/area  LOAs  and  directives . . 

© 

© 

35. 

Relevant  sector/area  special  procedures .  ' 

©I 

© 

© 

36. 

ATC  equipment  capabilities  and  limitations . 

© 

© 

© 

© 

37. 

Aircraft  types,  characteristics,  and  performance  limits . 

© 

© 

© 

'© 

38. 

Weather . . . . . 

© 

© 

'© 

39. 

Facility  general  policies  and  procedures .  ; 

;®; 

©' 

© 

© 

i©: 

40. 

Other  parts  of  the  ATC  system,  such  as  FSS  and  AF . 

®i 

©^ 

© 

41. 

FAA  organization,  general  policies,  and  procedures . 

© 

© 

© 

>© 

42. 

Pilot  roles,  responsibilities,  constraints,  and  workload . 

© 

© 

© 

S' 

t© 

Based  on  your  observations,  what  is  [Controller  namej’s  overall  potential  to 
succeed  in  the  ATC  occupation  (on  a  40-100  scale) . 

About  how  long  have  you  supervised  or  trained  [controller]?................ 


m  m 

(Yrs)  (Mths) 


Please  tell  us  a  little  about  yourself: 


About  how  long  have  you  been  an  air  traffic  controller?., 

About  how  long  have  you  been  at  this  facility? . 

About  how  long  have  you  been  in  your  present  position? 


Are  you  a .  O  Instructor  O  Supervisor  O  Manager 

THANK  YOU  VERY  MUCH! 
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Date 


[Controller] 

[Facility] 

[Mailing  address] 

[City,  ST,  Zip-xxxx] 

Dear  [controller] 

The  Collegiate  Training  Initiative  for  Air  Traffic  Control  Specialists  (CTI)  program  was  implemented  by  the  agency 
in  1990  in  order  to  assess  the  feasibility  of  developing  and  implementing  initial,  entry-level  ATCS  technical  training 
at  selected  educational  institutions  throughout  the  country.  As  a  graduate  from  one  of  the  five  participating  CTI 
schools,  you  were  placed  directly  into  a  field  facility  without  going  through  the  initial  training  at  the  FAA  Academy. 
This  survey  is  part  of  a  series  of  studies  being  conducted  by  the  Civil  Aeromedical  Institute  (CAMI)  to  evaluate  the 
feasibility  of  the  CTI  program.  These  survey  results  will  be  used  to  assess  how  well  the  CTI  programs  have  trained 
their  graduates  relative  to  FAA-trained  developmental. 

In  this  evaluation,  all  of  the  CTI  graduates  currently  employed  by  the  FAA  will  r^eive  this  CTI  Controller  Profile. 

In  the  survey,  you  are  asked  to  assess  your  own  performance  —  your  technical  skills,  your  technical  knowledge,  and 
teamwork  —  relative  to  all  other  controllers  you  have  known  at  the  same  point  in  their  career.  We  are  asking  you 
directly  because  the  research  literature  suggests  that  people  in  fact  can  and  do  give  very  honest  self-assessments 
when  asked.  People  know  their  strengths  and  where  they  need  training  or  practice. 

The  same  survey  is  being  sent  to  a  comparison  group  of  FAA  Academy-trained  controllers,  in  order  to  provide  a 
basis  for  comparing  how  well  the  CTI  and  Academy  programs  trained  new  controllers  for  success  in  field  training. 
Similar  surveys  are  also  being  sent  to  your  supervisor  and  senior  OJT  instructor  in  order  to  get  their  assessments  as 
well.  The  results  of  the  self-,  supervisor-,  and  instructor-assessments  will  be  combined  for  each  person,  and  then 
aggregated  within  groups  for  analysis.  Let  us  stress  to  you  that  the  focus  of  the  study  is  on  evaluating  the  CTI  and 
Academy  programs,  not  on  individuals. 

In  other  words,  only  group  statistics  shall  be  used  in  any  reports.  Data  about  individuals  are  confidential,  and  shall 
not  be  released.  The  data  collected  in  this  study  and  its  results  shall  not  be  used  in  any  way,  by  any  member, 
employee,  representative,  or  contractor,  of  the  agency  to  effect  your  assignments,  training,  working  conditions,  or 
status. 

Finally,  you  will  be  given  time  at  work  to  complete  this  survey.  The  profile  data  from  this  survey  from  each  and 
every  CTI  graduate  are  vital  to  evaluating  the  CTI  program  for  the  agency,  the  Department  of  Transportation,  and 
the  CTI  schools.  While  participation  is  voluntary,  we  strongly  urge  you  to  complete  the  survey  and  return  it  in  the 
postage  paid  envelope  as  soon  as  possible. 

/s/ 

[Name] 

[Air  Traffic  title] 


/s/ 

[Name] 
[AHT  title] 
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Collegiate  Training  Initiative  (CTI)  Graduate  Profile 
PURPOSE  OF  SURVEY 


This  survey  is  part  of  a  series  of  studies  to  evaluate  the  Collegiate  Trmning  Initiative  (CTI). 
[controller],  your  assessment  of  your  own  performance  is  very  important  to  determining  if  the 
CTI  program  is  accomplishing  the  goal  of  putting  high-aptitude  persons  into  the  field. 


BACKGROUND  OF  SURVEY 

The  CTI  program  was  initially  implemented  in  1990.  Our  records  indicate  that  you  graduated 
from  one  of  the  5  participating  institutions.  The  purpose  of  this  followup  study,  [controller],  is  to 
assess  the  performance  and  progress  of  CTI  graduates  as  part  of  the  overall  evaluation  of  the  CTI. 


THANKS  FOR  YOUR  COOPERATION 

The  profile  data  from  this  survey  are  vital  to  evaluating  the  CTI  for  the  agency.  Your 
cooperation  and  thoughtful  self-assessment  of  your  technical  skills,  teamwork,  and  knowledge  is 
greatly  appreciated  — and  needed. 


A  FINAL  WORD 

The  researchers  at  CAMI  know  that  it  seems  like  there  a  lot  of  surveys  in  the  FAA.  This  survey, 
however,  has  been  mailed  to  a  very  specific  group,  as  opposed  to  a  more  general  sample  like  the 
Job  Satisfaction  Survey.  We  also  know  that,  often,  not  much  seems  to  ever  be  heard  about  the 
results  of  a  survey.  To  combat  this  perception,  we’d  like  to  send  a  short  summary  of  the  results  to 
you,  when  they  become  available.  Just  fill  out  a  mailing  label  or  envelope,  and  include  it  in  the 
return  envelope. 


THANK  YOU  VERY  MUCH! 


A-9 


Please  remember  that  this  survey: 

•  Is  for  research  purposes  only  under  5  USC  1301, 2301,  &  3304; 

•  Shall  be  used  only  for  the  evaluation  of  the  CTI  by  CAMI  researchers; 

•  Is  confidential  and  only  group  statistics  shall  be  used  in  any  reports; 

•  Shall  not  be  discussed  with  your  OJTI,  supervisor,  or  any  other  person; 

•  Shall  not  effect  your  assignments,  working  conditions,  or  status;  and 

•  Shall  not  be  copied,  recorded,  or  retained  at  the  facility  for  any  purpose. 


TEAMWORK:  Consider  your  teamwork,  relative  to  all  other  controllers  you  have  known  at  the  same 
point  in  training.  Use  the  scale  below  to  profile  your  teamwork. _ _ 

®ssExceptionaUy _ ; _ ■  _ _ 


1  9  Acceptably  _ 

®  =  Mareinally 

{ 

•r  ®  SI  Can’t  say,  don't  know,  ornot  applicable 

! 

Relative  to  all  oflier  controllers  you  have  known  at  the  same  point 

T. 

in  their  career  as  you,  how  well  do  you: 

1 

WnrV  in  a  . . . 

w 

® 

0 

1 « 

2. 

Engage  other  team  members  in  solving  a  problem . 

0 

® 

3. 

Lead  the  team  in  solving  problems,  making  improvements,  etc. 

® 

0 

4. 

Earn  the  respect  of  team  members . 

® 

® 

0 

® 

5. 

Build  camadarie  or  spirit  appropriately  within  the  team . 

i® 

® 

0 

® 

6. 

Discourage  horse  play  or  other  disruptive  behavior . 

® 

11; 

0 

-® 

7. 

Support  or  aid  team  members  in  stressful  situations . 

© 

® 

0 

® 

8. 

Maintain  awareness  of  own  ability  limits . 

® 

® 

0 

'® 

9. 

Accept  feedback  regarding  performance . 

® 

0 

0 

10. 

Seek  additional  information  when  confronted  with  a  problem . 

'® 

® 

0 

0 

^® 

11. 

Evaluate  alternative  solutions  to  a  problem . 

® 

® 

0 

0 

® 

12. 

Perform  confidently . 

^® 

® 

0 

0 

® 

13. 

Perform  consistently  day  after  day  regardless  of  circumstances 

® 

® 

0 

0 

® 

14. 

Adapt  to  changing  conditions  or  circumstances  on  the  job . 

1© 

® 

'0 

0 

1® 

15. 

Tolerate  stressful  situations . 

© 

® 

0 

® 
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^  <S>  =  "Strmgty 

®  =  Very  much 

i?  0  a  Fo  some  degree 

®  =  Not  very  much 

?  ®  =  CanHsay,don*tknoWiOrnotappUcable 

Relative  to  all  other  controllers  you  have  known  at  the  same  point 

in  their  career  as  you,  to  what  degree  do  you  believe  that  you: 

16. 

Are  part  of  the  team . 

(S> 

© 

© 

© 

© 

17. 

Might  be  more  accepted  in  another  work  group  or  team . 

© 

© 

© 

© 

18. 

Fit  in  with  your  current  work  group  or  team . . . . . 

© 

© 

© 

© 

19. 

Are  treated  fairly  by  your  current  work  group  or  team . 

<§> 

© 

© 

© 

,© 

20. 

Are  affected  negatively  (with  regard  to  work  performance)  by  how 

the  other  controllers  on  your  current  team  view  you . 

© 

© 

© 

© 

SKILLS:  Now  consider  your  technical  skill  in  performing  air  traffic  control  tasks,  relative  to  all  other 
controllers  you  have  known  at  the  same  point  in  training.  Use  the  scale  below  to  profile  your  technical 
skills. 


/■:  ■ 

■  : 

\  Ji; 

©  =  Very  well 

III 

©  =  Marginally 

f'.: 

Relative  to  all  other  controllers  you  have  known  at  the  same  point 

in  their  career  as  you,  how  well  do  you: 

21. 

Ensure  separation  using  vectors,  speed,  &  altitude . 

© 

0 

© 

© 

22. 

Maintain  an  orderly  flow  of  traffic . 

© 

© 

© 

© 

© 

23. 

Sequence  traffic . . . 

© 

© 

© 

© 

© 

24. 

Perform  pointouts  and  handoffs . 

'© 

© 

© 

© 

© 

25. 

Manage  (surface)  traffic  movement  areas . 

;© 

© 

© 

© 

© 

26. 

Recover  from  and  correct  errors,  slips,  and/or  mistakes . 

;© 

© 

© 

© 

‘© 

27. 

Prioritize  actions . . . 

© 

© 

© 

© 

© 

28. 

Maintain  situational  awareness . 

© 

© 

© 

© 

© 

29. 

Issue  clearances  using  appropriate  phraseology . 

■© 

© 

® 

© 

© 

30. 

Conduct  relief  briefing . . . 

© 

© 

© 

© 

© 

31. 

Post  flight  data  on  flight  progress  strips . 

© 

1  ® 

© 
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KNOWLEDGE:  Finally,  consider  your  technical  knowledge  about  air  traffic  control,  relative  to  all 
other  controllers  you  have  known  at  the  same  point  in  training.  Use  the  scale  below  to  profile  your 
technical  knowledge:  _ _ _ 


SixMostevervthine 

d)  -  Much 

Q>=Some 

®  s:  Very  little 

«  Om?  sav,  don*t  know,  or  not  applicable 

Relative  to  all  other  controllers  you  have  observed  at  the  same 
point  in  their  career  as  you,  how  much  do  you  know  about: 

< 

I 

32. 

Airspace  configuration  in  sector  and/or  area  of  specialization 

(D 

'Q> 

0 

33. 

Traps,  hot  spots,  and  traffic  patterns  or  flows  in  the  sector/area 

;® 

® 

0 

34. 

Relevant  sector/area  LOAs  and  directives . 

[® 

0 

35. 

Relevant  sector/area  special  procedures . 

ii: 

0 

36. 

ATC  equipment  capabilities  and  limitations . 

w 

S'  ! 

G) 

0 

© 

37. 

Aircraft  types,  characteristics,  and  performance  limits . 

© 

0 

f  ■ 

j® 

© 

Q) 

0 

® 

JO. 

39. 

Facility  general  policies  and  procedures . 

1® 

© 

■0 

0 

40. 

Other  parts  of  the  ATC  system,  such  as  FSS  and  AF . 

^® 

1 

© 

0 

41. 

FAA  organization,  general  policies,  and  procedures . 

1® 

1 

© 

0 

■® 

Ii 

42. 

Pilot  roles,  responsibilities,  constraints,  and  workload . 

►® 

© 

^0 

0 

Based  on  your  self-assessment,  [controller],  what  is  your  overall  potential  to 
succeed  in  the  ATC  occupation  (on  a  40-100  scale) . 

About  how  long  did  you  have  to  wait  between  graduating  and  starting  the 
first  phase  of  field  training  at  your  facility? ... — .. — . — - ... — .... — ... — 


un 

mm 

(Yrs)  (Mths) 


THANK  YOU  VERY  MUCH! 


A-12 


Date 


[Supervisor] 

[Facility] 

[Mailing  address] 

[City,  ST,  Zip-xxxx] 

Dear  [supervisor] 

The  Collegiate  Training  Initiative  for  Air  Traffic  Control  Specialists  (CTI)  program  was  implemented  by  the  agency 
in  1990  in  order  to  assess  the  feasibility  of  developing  and  implementing  initial,  entry-level  ATCS  technical  training 
at  selected  educational  institutions  throughout  the  country.  [Controller]  graduated  from  one  of  the  five  participating 
CTI  schools,  and  was  placed  directly  into  a  field  facility  without  going  through  the  initial  training  at  the  FAA 
Academy.  This  survey  is  part  of  a  series  of  studies  being  conducted  by  the  Civil  Aeromedical  Institute  (CAMI)  to 
evaluate  the  feasibility  of  the  CTI  program.  These  survey  results  will  be  used  to  assess  how  well  the  CTI  programs 
have  trained  their  graduates  relative  to  FAA-trained  developmental. 

In  this  evaluation,  supervisors  of  and  selected  OJT  instructors  for  the  CTI  graduates  currently  employed  by  the  FAA 
will  receive  this  CTI  Controller  Profile.  In  the  survey,  you  are  asked  to  assess  the  performance  of  [controller] —  his 
or  her  technical  skills,  technical  knowledge,  and  teamwork  —  relative  to  all  other  controllers  you  have  known  at  the 
same  point  in  their  career. 

The  same  survey  is  being  sent  to  the  supervisors  of  and  OJT  instructors  for  a  comparison  group  of  FAA  Academy- 
trained  controllers,  in  order  to  provide  a  basis  for  comparing  how  well  the  CTI  and  Academy  programs  trained  new 
controllers  for  success  in  field  training.  Similar  surveys  are  also  being  sent  to  the  controllers  to  obtain  their  self- 
assessments  as  well.  The  results  of  the  self-,  supervisor-,  and  instructor-assessments  will  be  combined  for  each 
person,  and  then  aggregated  within  groups  for  analysis.  Let  us  stress  to  you  that  the  focus  of  the  study  is  on 
evaluating  the  CTI  and  Academy  programs. 

In  other  words,  only  group  statistics  shall  be  used  in  any  reports.  Data  about  individuals  are  confidential,  and  shall 
not  be  released.  The  data  collected  in  this  study  and  its  results  shall  not  be  used  in  any  way,  by  any  member, 
employee,  representative,  or  contractor,  of  the  agency  to  effect  your  assignments,  training,  working  conditions,  or 
status.  The  surveys  shall  not  be  retained,  recorded,  or  copied  in  any  way  at  the  facility  for  any  purpose. 

Finally,  the  profile  data  from  this  survey  are  vital  to  evaluating  the  CTI  program  for  the  agency,  the  Department  of 
Transportation,  and  the  CTI  schools.  We  strongly  urge  you  to  complete  the  survey  and  return  it  in  the  postage  paid 
envelope  as  soon  as  possible. 

/s/ 

[Name] 

[Air  Traffic  title] 
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Collegiate  Training  Initiative  (CTI)  Graduate  Profile 


PURPOSE  OF  SURVEY 


This  survey  is  part  of  a  series  of  studies  to  evaluate  the  Collegiate  Training  Initiative  (CTI). 
Your  first-hand  impressions  of  the  performance  of  [controller  (SSN)]  are  very  important  to 
determining  if  the  CTI  program  is  accomplishing  the  goal  of  putting  high-aptitude  persons  into 
the  field. 


BACKGROUND  OF  SURVEY 

The  CTI  program  was  initially  implemented  in  1990.  [controller]  graduated  from  one  of  the  5 
participating  institutions.  The  purpose  of  this  followup  study  is  to  assess  the  performance  and 
progress  of  [controller]  as  part  of  the  overall  evaluation  of  the  CTI. 

THANKS  FOR  YOUR  COOPERATION 

The  profile  data  from  this  survey  are  vital  to  evaluating  the  CTI  for  the  agency.  Your 
cooperation  and  thoughtful  consideration  of  [controller] ’s  skills,  teamwork,  and  knowledge  is 
greatly  appreciated  —  and  needed. 

A  FINAL  WORD 

The  researchers  at  CAMI  know  that  it  seems  like  there  a  lot  of  surveys  in  the  FAA.  This  survey, 
however,  has  been  mailed  to  a  very  specific  group,  as  opposed  to  a  more  general  sample  like  the 
Job  Satisfaction  Survey.  We  also  know  that,  often,  not  much  seems  to  ever  be  heard  about  the 
results  of  a  survey.  To  combat  this  perception,  we’d  like  to  send  a  short  summary  of  the  results  to 
you,  when  they  become  available.  Just  fill  out  a  mailing  label  or  envelope,  and  include  it  in  the 
return  envelope. 

THANK  YOU  VERY  MUCH! 
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Please  remember  that  this  survey: 


•  Is  for  research  purposes  only  under  5  USC  1301, 2301,  &  3304; 

•  Shall  be  used  only  for  the  evaluation  of  the  CTI  by  CAMI  researchers; 

•  Is  confidential  and  only  group  statistics  shall  be  used  in  any  reports; 

•  Shall  not  be  discussed  with  the  controller  or  any  other  person; 

•  Shall  not  effect  assignments,  working  conditions,  or  status  of  the  controller;  and 

•  Shall  not  be  copied,  recorded,  or  retained  at  the  facility  for  any  purpose. 


TEAMWORK:  Consider  the  controller’s  teamwork,  relative  to  all  other  controllers  you  have 
observed  at  the  same  point  in  training.  Use  the  scale  below  to  profile  [controller] ’s  teamwork. 


iiiiiii 

lillill 

!/■: 

(D  =  Very  well 

Q)  =  Marginally 

liilM 

Relative  to  all  other  controllers  you  have  observed  at  the  same 
point  in  their  career,  how  well  does  this  controller: 

i- 

1. 

Work  in  a  team . . . 

■ 

® 

2. 

Engage  other  team  members  in  solving  a  problem . 

'0 

0 

:® 

3. 

Lead  the  team  in  solving  problems,  making  improvements,  etc. 

:  ® 

0 

0 

® 

4. 

Earn  the  respect  of  team  members . 

® 

® 

0 

a-  ■ 

0 

® 

5. 

Build  camadarie  or  spirit  appropriately  within  the  team . 

0 

6. 

Discourage  horse  play  or  other  disruptive  behavior . 

® 

® 

7. 

Support  or  aid  team  members  in  stressful  situations . 

® 

0 

® 

® 

8. 

Maintain  awareness  of  own  ability  limits . 

® 

0 

® 

® 

9. 

Accept  feedback  regarding  performance . 

® 

1® 

® 

i  ® 

10. 

Seek  additional  information  when  confronted  with  a  problem . 

® 

® 

0 

® 

® 

11. 

Evaluate  alternative  solutions  to  a  problem . 

® 

® 

0 

® 

® 

12. 

Perform  confidently . 

® 

0 

® 

@ 

13. 

Perform  consistently  day  after  day  regardless  of  circumstances 

® 

® 

0 

® 

® 

14. 

Adapt  to  changing  conditions  or  circumstances  on  the  job . 

® 

® 

0 

® 

® 

15. 

Tolerate  stressful  situations . 

® 

® 

® 

® 
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^  Strongly 

®  =  Very  much 

To  some  degree 

®  =  Not  very  much 

ill  !  '^Cedii  say,  don’t  kddd,  dr  not  applicable 

1. 

Relative  to  all  other  controllers  you  have  observed  at  this  point  in 

> 

ills 

I-  . 

‘ 

their  career,  to  what  degree  do  you  believe  that  this  controller: 

k 

i  ■ 

16. 

Is  part  of  the  team . 

® 

K 

17. 

Might  be  more  accepted  in  another  work  group  or  team . 

;® 

® 

ii 

•(3) 

18. 

Fits  in  with  his  or  her  current  work  group  or  team . 

® 

(2) 

® 

19. 

Is  treated  fairly  by  h  sr  or  his  current  work  group  or  team . 

Ml 

® 

® 

20. 

Is  affected  negatively  (with  regard  to  work  performance)  by  how 

the  other  controllers  on  his  or  her  current  team  view  her/him . 

t 

© 

® 

'Q> 

® 

SKILLS:  Now  consider  the  controller’s  technical  skill  in  performing  air  traffic  control  tasks,  relative 
to  all  other  controllers  you  have  observed  at  the  same  point  in  their  career.  Use  the  scale  below  to 
profile  [controller] ’s  technical  skills. 


. 

®  =  Very  well 

1  ®  ss  Acceptably  '^'4 ..  ^ 

Ijlll 

®  =  Marginally 

i  .  ■  . 

t- 

Relative  to  all  other  controllers  you  have  observed  at  this  point  in 
their  career,  how  well  does  this  controller: 

.  ■  ■ 

i  ■ 

21. 

Ensure  separation  using  vectors,  speed,  &  altitude . 

® 

® 

■® 

® 

22. 

Maintain  an  orderly  flow  of  traffic . 

m 

® 

® 

i® 

23. 

Sequence  traffic . 

i® 

0 

® 

® 

24. 

Perform  pointouts  and  handoffs . 

® 

® 

® 

® 

25. 

Manage  (surface)  traffic  movement  areas . 

;® 

® 

® 

® 

® 

26. 

Recover  from  and  correct  errors,  slips,  and/or  mistakes . 

ii; 

® 

IS 

® 

® 

27. 

Prioritize  actions . 

® 

® 

;® 

® 

® 

28. 

Maintain  situational  awareness . 

;® 

i;. 

®i 

1® 

® 

® 

29. 

Issue  clearances  using  appropriate  phraseology . 

1® 

® 

»  :■ 

® 

^®' 

30. 

Conduct  relief  briefing . 

P 

® 

0 

® 

p 

31. 

Post  flight  data  on  flight  progress  strips . 

&> 

® 

0 

® 

® 
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KNOWLEDGE:  Finally,  consider  the  controller’s  technical  knowledge  about  air  traffic  control, 
relative  to  all  other  controllers  you  have  observed  at  the  same  point  in  training.  Use  the  scale  below  to 
profile  [controller] ’s  technical  knowledge; 


iliilli 

iiiiiii 

®  =  Much 

®  =  Very  little 

. 

Relative  to  all  other  controllers  you  have  observed  at  the  same 
point  in  their  career,  how  much  does  this  controller  know  about: 

32. 

Airspace  configuration  in  sector  and/or  area  of  specialization 

® 

0 

0 

33. 

Traps,  hot  spots,  and  traffic  patterns  or  flows  in  the  sector/area 

0 

0 

34. 

Relevant  sector/area  LOAs  and  directives . 

0 

0 

35. 

Relevant  sector/area  special  procedures . 

0 

0 

® 

36. 

ATC  equipment  capabilities  and  limitations . . . 

0 

0 

® 

37. 

Aircraft  types,  characteristics,  and  performance  limits . 

0 

0 

@ 

38. 

Weather . 

0 

0 

39. 

Facility  general  policies  and  procedures . . 

© 

® 

0 

0 

% 

40. 

Other  parts  of  the  ATC  system,  such  as  FSS  and  AF . 

© 

® 

0 

0 

41. 

FAA  organization,  general  policies,  and  procedures . 

© 

0 

0 

"0 

42. 

Pilot  roles,  responsibilities,  constraints,  and  workload . 

© 

® 

0 

0 

■0 

Based  on  your  observations,  what  is  [Controller  name]’s  overall  potential  to 
succeed  in  the  ATC  occupation  (on  a  40-100  scale) . 

About  how  long  have  you  supervised  or  trained  [controller]? . 


m  m 

(Yrs)  (Mths) 


Please  tell  us  a  little  about  yourself: 


About  how  long  have  you  been  an  air  traffic  controller?., 

About  how  long  have  you  been  at  this  facility? . 

About  how  long  have  you  been  in  your  present  position? 


Are  you  a .  O  Instructor  O  Supervisor  O  Manager 

THANK  YOU  VERY  MUCH! 
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Appendix  B 

Cost-Benefit  Analysis  Model  Worksheets 
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Table  B-1 

FY1991  MnATCTC  Cost  estimate  worksheet 


Cost  Category 

Description 

N 

Rate 

$ 

Direct 

FAA  Financial  Support 

1.00 

$6,650,000 

$6,650,000 

Direct 

Site  Visits 

6.00 

$500 

$3,000 

Direct 

Regional  Liaison  (GS-14/5) 

1.00 

$59,394 

$59,394 

Program 

AHT  Staff  (GS-14/5) 

0.25 

$59,394 

$14,849 

Program 

AAM  Psychologist  (GS-12/1) 

0.10 

$37,294 

$3,729 

Program 

Evaluation  Contract(s) 

0.00 

$0 

$0 

Program 

APN  Staff  (GS- 13/5) 

0.00 

$50,260 

$0 

Program 

AMH  Staff  (GS-7/2) 

0.00 

$21,724 

$0 

Program 

AMA  Staff  (GS-14/5) 

0.01 

$59,394 

$594 

Program 

ATZ  Staff  (GS-14/5) 

0.04 

$59,394 

$2,376 

Annual  Costs 

$6,733,942 

Cumulative  Costs 

$6,733,942 

Annual  Hires 

26 

Cumulative  Hires 

26 

Cost  per  Hire 

$258,998 

Table  B-2 

FY 1 992  MnATCTC  Cost  estimate  worksheet 

Cost  Category 

Description 

N 

Rate 

$ 

Direct 

FAA  Financial  Support 

1.00 

$2,000,000 

$2,000,000 

Direct 

Site  Visits 

1.00 

$500 

$500 

Direct 

Regional  Liaison  (GS-14/5) 

0.00 

$61,887 

$0 

E*rogram 

AHT  Staff  (GS-14/5) 

0.05 

$61,887 

$3,094 

Program 

AAM  Psychologist  (GS-12/2) 

0.10 

$40,156 

$4,016 

Program 

Evaluation  Contract(s) 

0.00 

$0 

$0 

Program 

APN  Staff  (GS-13/9) 

0.10 

$58,530 

$5,853 

Program 

AMH  Staff  (GS-7/3) 

0.00 

$23,366 

$0 

Program 

AMA  Staff  (GS-14/5) 

0.01 

$61,887 

$619 

Program 

ATZ  Staff  (GS-14/5) 

0.04 

$61,887 

$2,475 

Annual  Costs 

$2,016,557 

Cumulative  Costs 

$8,750,499 

Annual  Hires 

19 

Cumulative  Hires 

45 

Cost  per  Hire 

$194,456 

Table  B-3 

FY1993  MnATCTC  Cost  estimate  worksheet 


Cost  Category 

Description 

N 

Rate 

$ 

Direct 

FAA  Financial  Support 

1.00 

$2,000,000 

$2,000,000 

Direct 

Site  Visits 

1.00 

$500 

$500 

Direct 

Regional  Liaison  (GS-14/5) 

0.50 

$64,179 

$32,090 

Program 

AHT  Staff  (GS-14/5) 

0.04 

$64,179 

$2,567 

Program 

AAM  Psychologist  (GS-13/1) 

0.02 

$47,920 

$958 

Program 

Evaluation  Contract(s) 

0.20 

$97,000 

$19,400 

Program 

APN  Staff  (GS- 13/9) 

0.07 

$60,696 

$4,249 

Program 

AMH  Staff  (GS-7/4) 

0.12 

$24,988 

$2,999 

Program 

AMA  r:aff  (GS-14/5) 

0.01 

$64,179 

$642 

Program 

ATZ  Staff  (GS-14/5) 

0.01 

$64,179 

$642 

Annual  Costs 

$2,064,046 

Cumulative  Costs 

$10,814,545 

Annual  Hires 

78 

Cumulative  Hires 

123 

Cost  per  Hire 

$87,923 

Table  B-4 

FY1994  MnATCTC  Cost  estimate  worksheet 

Cost  Category 

Description 

N 

Rate 

$ 

Direct 

FAA  Financial  Support 

1.00 

$0 

$0 

Direct 

Site  Visits 

1.00 

$500 

$500 

Direct 

Regional  Liaison  (GS-14/5) 

O.OI 

$64,179 

$642 

Program 

AHT  Staff  (GS- 14/6)* 

0.10 

$68,862 

$6,886 

Program 

AAM  Psychologist  (GS-13/2) 

0.10 

$51,171 

$5,117 

Program 

Evaluation  Contract(s) 

0.00 

$0 

$0 

Program 

APN  Staff  (GS-13/10)* 

0.10 

$64,928 

$6,493 

Program 

AMH  Staff  (GS-7/4) 

0.12 

$25,823 

$3,099 

Program 

AMA  Staff  (GS-14/5) 

0.01 

$66,323 

$663 

Program 

ATZ  Staff  (GS-14/5)* 

0.02 

$66,323 

$1,326 

^Locality  pay 

Annual  Costs 

$24,726 

Cumulative  Costs 

$10,839,271 

Annual  Hires 

40 

Cumulative  Hires 

163 

Cost  per  Hire 

$66,499 

B-4 


Table  B-5 

FY199S  MnATCTC  Cost  estimate  worksheet 


Cost  Category 

Description 

N 

Rate 

$ 

Direct 

FAA  Financial  Support 

1.00 

$0 

$0 

Direct 

Site  Visits 

0.00 

$2,000 

$0 

Direct 

Regional  Liaison  (GS-14/5) 

0.01 

$65,460 

$655 

Program 

AHR-15  Staff  (GS-14/6)* 

0.02 

$71,078 

$1,422 

Program 

AAM  Psychologist  (GS-13/3)* 

0.10 

$52,136 

$5,214 

Program 

Evaluation  Contract(s) 

0.00 

$0 

$0 

Program 

AHR-22  Staff  (GS-13/10)* 

0.05 

$67,021 

$3,351 

Program 

AMH  Staff  (GS-7/5)* 

0.12 

$26,529 

$3,183 

Program 

AMA  Staff  (GS-14/5)* 

0.01 

$65,460 

$655 

Program 

ATZSTaff  (GS-14/5)* 

0.02 

$69,047 

$1,381 

^Locality  pay 

Annual  Costs 

$15,860 

Cumulative  Costs 

$10,855,131 

Annual  Hires 

30 

Cumulative  Hires 

193 

Cost  per  Hire 

$56,244 

Table  B-6 

FY1996  MnATCTC  Cost  estimate  worksheet 

Cost  Category 

Description 

N 

Rate 

$ 

Direct 

FAA  Financial  Support 

1.00 

$250,000 

$250,000 

Direct 

Site  Visits 

0.00 

$0 

$0 

Direct 

Regional  Liaison  (GS-14/5) 

0.01 

$65,460 

$655 

Program 

AHR-15  Staff  (GS-14/6)* 

0.05 

$71,078 

$3,554 

Program 

AAM  Psychologist  (GS-14/1)* 

0.10 

$57,760 

$5,776 

Program 

Evaluation  Contract(s) 

0.20 

$0 

$0 

Program 

AHR-22  Staff  (GS-13/10)* 

0.05 

$67,021 

$3,351 

Program 

AMH  Staff  (GS-7/5)* 

0.12 

$26,529 

$3,183 

Program 

AMA  Staff  (GS-14/5)* 

0.01 

$65,460 

$655 

Program 

ATZ  Staff  (GS-14/5)* 

0.02 

$69,047 

$1,381 

"^Locality  pay 

Annual  Costs 

$268,555 

Cumulative  Costs 

$11,123,686 

Annual  Hires 

0 

Cumulative  Hires 

193 

Cost  per  Hire 

$57,636 

B-5 


Table  B-7 

FY1997  MnATCTC  Cost  estimate  worksheet 


Cost  Category 

Description 

N 

Rate 

$ 

Direct 

FAA  Financial  Support 

1.00 

$1,700,000 

$1,700,000 

Direct 

Site  Visits 

0.00 

$0 

$0 

Direct 

Regional  Liaison  (GS-14/5) 

0.01 

$65,460 

$655 

Program 

AHR-15  Staff  (GS-1477)* 

0.05 

$73,108 

$3,655 

Program 

AAM  Psychologist  (GS-14/2)* 

0.10 

$59,685 

$5,969 

Program 

Evaluation  Contract(s) 

0.20 

$0 

$0 

Program 

AHR-22  Staff  (GS-13/10)* 

0.05 

$67,021 

$3,351 

Program 

AMH  Staff  (GS-7/6)* 

0.12 

$27,031 

$3,244 

Program 

AMA  :.iaff  (GS-14/6)* 

0.01 

$67,385 

$674 

Program 

ATZ  Staff  (GS-14/6)* 

0.02 

$71,078 

$1,422 

^Locality  pay 

Annual  Costs 

$1,718,969 

Cumulative  Costs 

$12,842,654 

Annual  Hires 

64 

Cumulative  Hires 

257 

Cost  per  Hire 

$49,971 

Table  B-8 

FY1998  MnATCTC  Cost  estimate  worksheet 

Cost  Category  Description 

N 

Rate 

$ 

Direct 

FAA  Financial  Support 

1.00 

$1,500,000 

$1,500,000 

Direct 

Site  Visits 

0.00 

$0 

$0 

Direct 

Regional  Liaison  (GS-14/5) 

0.01 

$65,460 

$655 

Program 

AHR-15  Staff  (GS-1477)* 

0.05 

$73,108 

$3,655 

Program 

AAM  Psychologist  (GS-14/2)* 

0.10 

$59,685 

$5,969 

Program 

Evaluation  Contract(s) 

0.20 

$0 

$0 

Program 

AHR-22  Staff  (GS-13/10)* 

0.05 

$67,021 

$3,351 

Program 

AMH  Staff  (GS-7/6)* 

0.12 

$27,031 

$3,244 

Program 

AMA  Staff  (GS- 14/6)* 

0.01 

$67,385 

$674 

Program 

ATZ  Staff  (GS-14/6)* 

0.02 

$71,078 

$1,422 

^Locality  pay 

Annual  Costs 

$1,518,969 

Cumulative  Costs 

$14,361,623 

Annual  Hires 

100 

Cumulative  Hires 

357 

Cost  per  Hire 

$40,229 

B-6 


Table  B-9 

FY1999  MnATCTC  Cost  estimate  worksheet 


Cost  Category 

Description 

N 

Rate 

$ 

Direct 

FAA  Financial  Support 

1.00 

$1,500,000 

$1,500,000 

Direct 

Site  Visits 

0.00 

$500 

$0 

Direct 

Regional  Liaison  (GS-14/5) 

0.01 

$65,460 

$655 

Program 

AHR-15  Staff  (GS-14/8)* 

0.05 

$75,139 

$3,757 

Program 

AAM  Psychologist  (GS-14/3)* 

0.10 

$61,610 

$6,161 

Program 

Evaluation  Contract(s) 

0.20 

$0 

$0 

Program 

AHR-22  Staff  (GS-13/10)* 

0.05 

$67,021 

$3,351 

Program 

AMH  Staff  (GS-7/7)* 

0.12 

$27,803 

$3,336 

Program 

AMA  Staff  (GS-14/7)* 

0.01 

$69,310 

$693 

Program 

ATZ  Staff  (GS-14/7)* 

0.02 

$73,108 

$1,462 

^Locality  pay 

Annual  Costs 

$1,519,415 

Cumulative  Costs 

$15,881,038 

Annual  Hires 

100 

Cumulative  Hires 

457 

Cost  per  Hire 

$34,751 

Table  B-IO 

FY2000  MnATCTC  Cost  estimate  worksheet 

Cost  Category 

Description 

N 

Rate 

$ 

Direct 

FAA  Financial  Support 

1.00 

$1,500,000 

$1,500,000 

Direct 

Site  Visits 

0.00 

$500 

$0 

Direct 

Regional  Liaison  (GS-14/5) 

0.01 

$65,460 

$655 

Program 

AHR-15  Staff  (GS-14/8)* 

0.05 

$75,139 

$3,757 

Program 

AAM  Psychologist  (GS-14/3)* 

0.10 

$61,610 

$6,161 

Program 

Evaluation  Contract(s) 

0.20 

$0 

$0 

Program 

AHR-22  Staff  (GS-13/10)* 

0.05 

$67,021 

$3,351 

Program 

AMH  Staff  (GS-7/7)* 

0.12 

$27,803 

$3,336 

Program 

AMA  Staff  (GS-14/7)* 

0.01 

$69,310 

$693 

Program 

ATZ  Staff  (GS-14/7)* 

0.02 

$73,108 

$1,462 

^Locality  pay 

Annual  Costs 

$1,519,415 

Cumulative  Costs 

$17,400,453 

Annual  Hires 

100 

Cumulative  Hires 

557 

Cost  per  Hire 

$31,240 

B-7 


Table  B-11 

Fy2001  MnATCTC  Cost  estimate -worksheet 


Cost  Category 

Description 

N 

Rate 

$ 

Direct 

FAA  Rnancial  Support 

1.00 

$1,500,000 

$1,500,000 

Direct 

Site  Visits 

0.00 

$500 

$0 

Direct 

Regional  Liaison  (GS-14/5) 

0.01 

$65,460 

$655 

Program 

AHR-15  Staff  (GS-14/8)* 

0.05 

$75,139 

$3,757 

Program 

AAM  Psychologist  (GS- 14/3)* 

0.10 

$61,610 

$6,161 

Program 

Evaluation  Contract(s) 

0.20 

$0 

$0 

Program 

AHR-22  Staff  (GS-13/10)* 

0.05 

$67,021 

$3,351 

Program 

AMH  Staff  (GS-7/7)* 

0.12 

$27,803 

$3,336 

Program 

AMA  Staff  (GS-14/7)* 

0.01 

$69,310 

$693 

Program 

ATZ  Staff  (GS- 14/7)* 

0.02 

$73,108 

$1,462 

^Locality  pay 

Annual  Costs 

$1,519,415 

Cumulative  Costs 

$18,919,869 

Annual  Hires 

100 

Cumulative  Hires 

657 

Cost  per  Hire 

$28,797 

Table  B-12 

Fy2002  MnATCTC  Cost  estimate  worksheet 

Cost  Category  Description 

N 

Rate 

,  $ 

Direct 

FAA  Financial  Support 

1.00 

$1,500,000 

$1,500,000 

Direct 

Site  Visits 

0.00 

$500 

.  $0 

Direct 

Regional  Liaison  (GS-14/5) 

0.01 

$65,460 

$655 

Program 

AHR-15  Staff  (GS-14/8)* 

0.05 

$75,139 

$3,757 

Program 

AAM  Psychologist  (GS-14/3)* 

0.10 

$61,610 

$6,161 

Program 

Evaluation  Contract(s) 

0.20 

$0 

$0 

Program 

AHR-22  Staff  (GS-13/10)* 

0.05 

$67,021 

$3,351 

Program 

AMH  Staff  (GS-7/7)* 

0.12 

$27,803 

$3,336 

Program 

AMA  Staff  (GS-14/7)* 

0.01 

$69,310 

$693 

Program 

ATZ  Staff  (GS-14/7)* 

0.02 

$73,108 

$1,462 

^Locality  pay 

Annual  Costs 

$1,519,415 

Cumulative  Costs 

$20,439,284 

Annual  Hires 

100 

Cumulative  Hires 

757 

Cost  per  Hire 

$27,000 

Table  B-13 

FY2003  MnATCTC  Cost  estimate  worksheet 


Cost  Category 

Description 

N 

Rate 

$ 

Direct 

FAA  Financial  Support 

1.00 

$1,500,000 

$1,500,000 

Direct 

Site  Visits 

0.00 

$500 

$0 

Direct 

Regional  Liaison  (GS-14/5) 

0.01 

$65,460 

$655 

Program 

AHR-15  Staff  (GS-14/8)* 

0.05 

$75,139 

$3,757 

Program 

AAM  Psychologist  (GS-14/3)* 

0.10 

$61,610 

$6,161 

Program 

Evaluation  Contract(s) 

0.20 

$0 

$0 

Program 

AHR-22  Staff  (GS-13/10)* 

0.05 

$67,021 

$3,351 

Program 

AMH  Staff  (GS-7/7)* 

0.12 

$27,803 

$3,336 

Program 

AMA  Staff  (GS-1477)* 

0.01 

$69,310 

$693 

Program 

ATZ  Staff  (GS-14/7)* 

0.02 

$73,108 

$1,462 

"^Locality  pay 

Annual  Costs 

$1,519,415 

Cumulative  Costs 

$21,958,699 

Annual  Hires 

100 

Cumulative  Hires 

857 

Cost  per  Hire  $25,623 


MnATCTC  hires  by  FAA  FAA  Screening _ _ FAA  Academy _ - 

Cumulative  Academics  Techniques  Skills-building  Cumulative 

Annual  Cumulative  Screening  cost  avoided  costs  (Phase  I)  cost  (Phase  H)  cost  (Phase  HI)  cost  avoided  costs 
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